You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Allan Folting <af...@gmail.com> on 2023/02/22 20:59:33 UTC
[DISCUSS] Show Python code examples first in Spark documentation
Hi all,
I would like to propose that we show Python code examples first in the
Spark documentation where we have multiple programming language examples.
An example is on the Quick Start page:
https://spark.apache.org/docs/latest/quick-start.html
I propose this change because Python has become more popular than the other
languages supported in Apache Spark. There are a lot more users of Spark in
Python than Scala today and Python attracts a broader set of new users.
For Python usage data, see https://www.tiobe.com/tiobe-index/ and
https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
Also, this change aligns with Python already being the first tab on our
home page:
https://spark.apache.org/
Anyone who wants to use another language can still just click on the other
tabs.
I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page
as a first step:
https://github.com/apache/spark/pull/40087
I would appreciate it if you could share your thoughts on this proposal.
Thanks a lot,
Allan Folting
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Mich Talebzadeh <mi...@gmail.com>.
I have seen many data engineering teams start out with Scala because
technically it is the best choice for many given reasons and basically it
is what Spark is. I also concur that Python is more popular than Scala
because of the advent of data science. A majority of use cases we see these
days are data science or related use cases where people mostly do Python.
Most Cloud Data warehouses offer embedded modeling tools that rely
extensively on Python packages. So, if you need those two worlds to
share code e and even handover code, you do not want the ideological battle
of Scala vs Python. Often we chose python for the sake of everybody
speaking the same language.
With regard to Spark docs showing Python code in the first tab etc, in my
opinion it is a moot point. The laws of diminishing return may indicate
that the time spent on changing the order may not worth it
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 22 Feb 2023 at 21:00, Allan Folting <af...@gmail.com> wrote:
> Hi all,
>
> I would like to propose that we show Python code examples first in the
> Spark documentation where we have multiple programming language examples.
> An example is on the Quick Start page:
> https://spark.apache.org/docs/latest/quick-start.html
>
> I propose this change because Python has become more popular than the
> other languages supported in Apache Spark. There are a lot more users of
> Spark in Python than Scala today and Python attracts a broader set of new
> users.
> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
>
> Also, this change aligns with Python already being the first tab on our
> home page:
> https://spark.apache.org/
>
> Anyone who wants to use another language can still just click on the other
> tabs.
>
> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page
> as a first step:
> https://github.com/apache/spark/pull/40087
>
>
> I would appreciate it if you could share your thoughts on this proposal.
>
>
> Thanks a lot,
> Allan Folting
>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Sean Owen <sr...@gmail.com>.
FWIW I agree with this.
On Wed, Feb 22, 2023 at 2:59 PM Allan Folting <af...@gmail.com> wrote:
> Hi all,
>
> I would like to propose that we show Python code examples first in the
> Spark documentation where we have multiple programming language examples.
> An example is on the Quick Start page:
> https://spark.apache.org/docs/latest/quick-start.html
>
> I propose this change because Python has become more popular than the
> other languages supported in Apache Spark. There are a lot more users of
> Spark in Python than Scala today and Python attracts a broader set of new
> users.
> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
>
> Also, this change aligns with Python already being the first tab on our
> home page:
> https://spark.apache.org/
>
> Anyone who wants to use another language can still just click on the other
> tabs.
>
> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page
> as a first step:
> https://github.com/apache/spark/pull/40087
>
>
> I would appreciate it if you could share your thoughts on this proposal.
>
>
> Thanks a lot,
> Allan Folting
>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Matei Zaharia <ma...@gmail.com>.
I’m +1 on switching to Python by default given what I see at the majority of users. I like the idea of investigating a way to save the language choice in a cookie and to switch all code examples on the page to a new language when you click one of the tabs. We used to have the switching behavior at least (e.g. see this archived page from 2016 https://web.archive.org/web/20160308055505/https://spark.apache.org/docs/latest/quick-start.html <https://web.archive.org/web/20160308055505/https://spark.apache.org/docs/latest/quick-start.html>), so I’m not sure what happened to that. We might never have had the cookie, but that is worth investigating.
Matei
> On Feb 23, 2023, at 11:31 PM, Santosh Pingale <sa...@adyen.com.invalid> wrote:
>
> Yes, I definitely agree and +1 to the proposal (FWIW).
>
> I was looking at Dongjoon's comments which made a lot of sense to me and trying to come up with an approach that provides smooth segway to python as first tab later on. But this is mostly guess work as I do not personally know the actual user behaviour on docs site.
>
> On Fri, Feb 24, 2023, 8:01 AM Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
> That sounds good to have that especially given that it will allow more flexibility to the users.
> But I think that's slightly orthogonal to this proposal since this proposal is more about the default (before users take an action).
>
>
> On Fri, 24 Feb 2023 at 15:35, Santosh Pingale <santosh.pingale@adyen.com <ma...@adyen.com>> wrote:
> Very interesting and user focused discussion, thanks for the proposal.
>
> Would it be better if we rather let users set the preference about the language they want to see first in the code examples? This preference can be easily stored on the browser side and used to decide ordering. This is inline with freedom users have with spark today.
>
>
> On Fri, Feb 24, 2023, 4:46 AM Allan Folting <afolting.dm@gmail.com <ma...@gmail.com>> wrote:
> I think this needs to be consistently done on all relevant pages and my intent is to do that work in time for when it is first released.
> I started with the "Spark SQL, DataFrames and Datasets Guide" page to break it up into multiple, scoped PRs.
> I should have made that clear before.
>
> I think it's a great idea to have an umbrella JIRA for this to outline the full scope and track overall progress and I'm happy to create it.
>
> I can't speak on behalf of all Scala users of course, but I don't think this change makes Scala appear as a 2nd class citizen, like I don't think of Python as a 2nd class citizen because it is not first currently, but it does recognize that Python is more broadly popular today.
>
> Thanks,
> Allan
>
> On Thu, Feb 23, 2023 at 6:55 PM Dongjoon Hyun <dongjoon.hyun@gmail.com <ma...@gmail.com>> wrote:
> Thank you all.
>
> Yes, attracting more Python users and being more Python user-friendly is always good.
>
> Basically, SPARK-42493 is proposing to introduce intentional inconsistency to Apache Spark documentation.
>
> The inconsistency from SPARK-42493 might give Python users the following questions first.
>
> - Why not RDD pages which are the heart of Apache Spark? Is Python not good in RDD?
> - Why not ML and Structured Streaming pages when DATA+AI Summit focuses on ML heavily?
>
> Also, more questions to the Scala users.
> - Is Scala language stepping down to the 2nd citizen language?
> - What about Scala 3?
>
> Of course, I understand SPARK-42493 has specific scopes (SQL/Dataset/Dataframe) and didn't mean anything like the above at all.
> However, if SPARK-42493 is emphasized as "the first step" to introduce that inconsistency, I'm wondering
> - What direction we are heading?
> - What is the next target scope?
> - When it will be achieved (or completed)?
> - Or, is the goal to be permanently inconsistent in terms of the documentation?
>
> It's unclear even in the documentation-only scope. If we are expecting more and more subtasks during Apache Spark 3.5 timeframe, shall we have an umbrella JIRA?
>
> Bests,
> Dongjoon.
>
>
> On Thu, Feb 23, 2023 at 6:15 PM Allan Folting <afolting.dm@gmail.com <ma...@gmail.com>> wrote:
> Thanks a lot for the questions and comments/feedback!
>
> To address your questions Dongjoon, I do not intend for these updates to the documentation to be tied to the potential changes/suggestions you ask about.
>
> In other words, this proposal is only about adjusting the documentation to target the majority of people reading it - namely the large and growing number of Python users - and new users in particular as they are often already familiar with and have a preference for Python when evaluating or starting to use Spark.
>
> While we may want to strengthen support for Python in other ways, I think such efforts should be tracked separately from this.
>
> Allan
>
> On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <mich.talebzadeh@gmail.com <ma...@gmail.com>> wrote:
> If this is not just flip flopping the document pages and involves other changes, then a proper impact analysis needs to be done to assess the efforts involved. Personally I don't think it really matters.
>
> HTH
>
>
> view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
> https://en.everybodywiki.com/Mich_Talebzadeh <https://en.everybodywiki.com/Mich_Talebzadeh>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
>
>
>
> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
> > 1. Does this suggestion imply Python API implementation will be the new blocker in the future in terms of feature parity among languages? Until now, Python API feature parity was one of the audit items because it's not enforced. In other words, Scala and Java have been the full feature because they are the underlying main developer languages while Python/R/SQL environments were the nice-to-have.
>
> I think it wouldn't be treated as a blocker .. but I do believe we have added all new features into the Python side for the last couple of releases. So, I wouldn't worry about this at this moment - we have been doing fine in terms of feature parity.
>
> > 2. Does this suggestion assume that the Python environment is easier for users than Scala/Java always? Given that we support Python 3.8 to 3.11, the support matrix for Python library dependency is a problem for the Apache Spark community to solve in order to claim that. As we say at SPARK-41454, Python language also introduces breaking changes to us historically and we have many `Pinned` python libraries issues.
>
> Yes. In fact, regardless of this change, I do believe we should test more versions, etc. At least scheduled jobs like we're doing JDK and Scala versions.
>
>
> FWIW, my take about this change is: people use Python and PySpark more (according to the chart and stats provided) so let's put those examples first :-).
>
>
> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <dongjoon.hyun@gmail.com <ma...@gmail.com>> wrote:
> I have two questions to clarify the scope and boundaries.
>
> 1. Does this suggestion imply Python API implementation will be the new blocker in the future in terms of feature parity among languages? Until now, Python API feature parity was one of the audit items because it's not enforced. In other words, Scala and Java have been the full feature because they are the underlying main developer languages while Python/R/SQL environments were the nice-to-have.
>
> 2. Does this suggestion assume that the Python environment is easier for users than Scala/Java always? Given that we support Python 3.8 to 3.11, the support matrix for Python library dependency is a problem for the Apache Spark community to solve in order to claim that. As we say at SPARK-41454, Python language also introduces breaking changes to us historically and we have many `Pinned` python libraries issues.
>
> Changing documentation is easy, but I hope we can give clear communication and direction in this effort because this is one of the most user-facing changes.
>
> Dongjoon.
>
> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ma...@qq.com> <ruifengz@foxmail.com <ma...@foxmail.com>> wrote:
> +1 LGTM
>
>
> Ruifeng Zheng
> ruifengz@foxmail.com
> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>
>
>
> ------------------ Original ------------------
> From: "Xinrong Meng" <xinrong.apache@gmail.com <ma...@gmail.com>>;
> Date: Thu, Feb 23, 2023 09:17 AM
> To: "Allan Folting"<afolting.dm@gmail.com <ma...@gmail.com>>;
> Cc: "dev"<dev@spark.apache.org <ma...@spark.apache.org>>;
> Subject: Re: [DISCUSS] Show Python code examples first in Spark documentation
>
> +1 Good idea!
>
> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <jackagoodson@gmail.com <ma...@gmail.com>> wrote:
> Good idea, at the company I work at we discussed using Scala as our primary language because technically it is slightly stronger than python but ultimately chose python in the end as it’s easier for other devs to be on boarded to our platform and future hiring for the team etc would be easier
>
> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
> +1 I like this idea too.
>
> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <afolting.dm@gmail.com <ma...@gmail.com>> wrote:
> Hi all,
>
> I would like to propose that we show Python code examples first in the Spark documentation where we have multiple programming language examples.
> An example is on the Quick Start page:
> https://spark.apache.org/docs/latest/quick-start.html <https://spark.apache.org/docs/latest/quick-start.html>
>
> I propose this change because Python has become more popular than the other languages supported in Apache Spark. There are a lot more users of Spark in Python than Scala today and Python attracts a broader set of new users.
> For Python usage data, see https://www.tiobe.com/tiobe-index/ <https://www.tiobe.com/tiobe-index/> and https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava <https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava>.
>
> Also, this change aligns with Python already being the first tab on our home page:
> https://spark.apache.org/ <https://spark.apache.org/>
>
> Anyone who wants to use another language can still just click on the other tabs.
>
> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page as a first step:
> https://github.com/apache/spark/pull/40087 <https://github.com/apache/spark/pull/40087>
>
>
> I would appreciate it if you could share your thoughts on this proposal.
>
>
> Thanks a lot,
> Allan Folting
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Santosh Pingale <sa...@adyen.com.INVALID>.
Yes, I definitely agree and +1 to the proposal (FWIW).
I was looking at Dongjoon's comments which made a lot of sense to me and
trying to come up with an approach that provides smooth segway to python as
first tab later on. But this is mostly guess work as I do not personally
know the actual user behaviour on docs site.
On Fri, Feb 24, 2023, 8:01 AM Hyukjin Kwon <gu...@gmail.com> wrote:
> That sounds good to have that especially given that it will allow more
> flexibility to the users.
> But I think that's slightly orthogonal to this proposal since this
> proposal is more about the default (before users take an action).
>
>
> On Fri, 24 Feb 2023 at 15:35, Santosh Pingale <sa...@adyen.com>
> wrote:
>
>> Very interesting and user focused discussion, thanks for the proposal.
>>
>> Would it be better if we rather let users set the preference about the
>> language they want to see first in the code examples? This preference can
>> be easily stored on the browser side and used to decide ordering. This is
>> inline with freedom users have with spark today.
>>
>>
>> On Fri, Feb 24, 2023, 4:46 AM Allan Folting <af...@gmail.com>
>> wrote:
>>
>>> I think this needs to be consistently done on all relevant pages and my
>>> intent is to do that work in time for when it is first released.
>>> I started with the "Spark SQL, DataFrames and Datasets Guide" page to
>>> break it up into multiple, scoped PRs.
>>> I should have made that clear before.
>>>
>>> I think it's a great idea to have an umbrella JIRA for this to outline
>>> the full scope and track overall progress and I'm happy to create it.
>>>
>>> I can't speak on behalf of all Scala users of course, but I don't think
>>> this change makes Scala appear as a 2nd class citizen, like I don't think
>>> of Python as a 2nd class citizen because it is not first currently, but it
>>> does recognize that Python is more broadly popular today.
>>>
>>> Thanks,
>>> Allan
>>>
>>> On Thu, Feb 23, 2023 at 6:55 PM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> Thank you all.
>>>>
>>>> Yes, attracting more Python users and being more Python user-friendly
>>>> is always good.
>>>>
>>>> Basically, SPARK-42493 is proposing to introduce intentional
>>>> inconsistency to Apache Spark documentation.
>>>>
>>>> The inconsistency from SPARK-42493 might give Python users the
>>>> following questions first.
>>>>
>>>> - Why not RDD pages which are the heart of Apache Spark? Is Python not
>>>> good in RDD?
>>>> - Why not ML and Structured Streaming pages when DATA+AI Summit focuses
>>>> on ML heavily?
>>>>
>>>> Also, more questions to the Scala users.
>>>> - Is Scala language stepping down to the 2nd citizen language?
>>>> - What about Scala 3?
>>>>
>>>> Of course, I understand SPARK-42493 has specific scopes
>>>> (SQL/Dataset/Dataframe) and didn't mean anything like the above at all.
>>>> However, if SPARK-42493 is emphasized as "the first step" to introduce
>>>> that inconsistency, I'm wondering
>>>> - What direction we are heading?
>>>> - What is the next target scope?
>>>> - When it will be achieved (or completed)?
>>>> - Or, is the goal to be permanently inconsistent in terms of the
>>>> documentation?
>>>>
>>>> It's unclear even in the documentation-only scope. If we are expecting
>>>> more and more subtasks during Apache Spark 3.5 timeframe, shall we have an
>>>> umbrella JIRA?
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>>
>>>> On Thu, Feb 23, 2023 at 6:15 PM Allan Folting <af...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks a lot for the questions and comments/feedback!
>>>>>
>>>>> To address your questions Dongjoon, I do not intend for these updates
>>>>> to the documentation to be tied to the potential changes/suggestions you
>>>>> ask about.
>>>>>
>>>>> In other words, this proposal is only about adjusting the
>>>>> documentation to target the majority of people reading it - namely the
>>>>> large and growing number of Python users - and new users in particular as
>>>>> they are often already familiar with and have a preference for Python when
>>>>> evaluating or starting to use Spark.
>>>>>
>>>>> While we may want to strengthen support for Python in other ways, I
>>>>> think such efforts should be tracked separately from this.
>>>>>
>>>>> Allan
>>>>>
>>>>> On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> If this is not just flip flopping the document pages and involves
>>>>>> other changes, then a proper impact analysis needs to be done to assess the
>>>>>> efforts involved. Personally I don't think it really matters.
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> > 1. Does this suggestion imply Python API implementation will be
>>>>>>> the new blocker in the future in terms of feature parity among languages?
>>>>>>> Until now, Python API feature parity was one of the audit items because
>>>>>>> it's not enforced. In other words, Scala and Java have been the full
>>>>>>> feature because they are the underlying main developer languages while
>>>>>>> Python/R/SQL environments were the nice-to-have.
>>>>>>>
>>>>>>> I think it wouldn't be treated as a blocker .. but I do believe we
>>>>>>> have added all new features into the Python side for the last couple of
>>>>>>> releases. So, I wouldn't worry about this at this moment - we have been
>>>>>>> doing fine in terms of feature parity.
>>>>>>>
>>>>>>> > 2. Does this suggestion assume that the Python environment is
>>>>>>> easier for users than Scala/Java always? Given that we support Python 3.8
>>>>>>> to 3.11, the support matrix for Python library dependency is a problem for
>>>>>>> the Apache Spark community to solve in order to claim that. As we say
>>>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>>>
>>>>>>> Yes. In fact, regardless of this change, I do believe we should test
>>>>>>> more versions, etc. At least scheduled jobs like we're doing JDK and Scala
>>>>>>> versions.
>>>>>>>
>>>>>>>
>>>>>>> FWIW, my take about this change is: people use Python and PySpark
>>>>>>> more (according to the chart and stats provided) so let's put those
>>>>>>> examples first :-).
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have two questions to clarify the scope and boundaries.
>>>>>>>>
>>>>>>>> 1. Does this suggestion imply Python API implementation will be the
>>>>>>>> new blocker in the future in terms of feature parity among languages? Until
>>>>>>>> now, Python API feature parity was one of the audit items because it's not
>>>>>>>> enforced. In other words, Scala and Java have been the full feature because
>>>>>>>> they are the underlying main developer languages while Python/R/SQL
>>>>>>>> environments were the nice-to-have.
>>>>>>>>
>>>>>>>> 2. Does this suggestion assume that the Python environment is
>>>>>>>> easier for users than Scala/Java always? Given that we support Python 3.8
>>>>>>>> to 3.11, the support matrix for Python library dependency is a problem for
>>>>>>>> the Apache Spark community to solve in order to claim that. As we say
>>>>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>>>>
>>>>>>>> Changing documentation is easy, but I hope we can give clear
>>>>>>>> communication and direction in this effort because this is one of the most
>>>>>>>> user-facing changes.
>>>>>>>>
>>>>>>>> Dongjoon.
>>>>>>>>
>>>>>>>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <
>>>>>>>> ruifengz@foxmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1 LGTM
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>> Ruifeng Zheng
>>>>>>>>> ruifengz@foxmail.com
>>>>>>>>>
>>>>>>>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------ Original ------------------
>>>>>>>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>>>>>>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>>>>>>>> *To:* "Allan Folting"<af...@gmail.com>;
>>>>>>>>> *Cc:* "dev"<de...@spark.apache.org>;
>>>>>>>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>>>>>>>> documentation
>>>>>>>>>
>>>>>>>>> +1 Good idea!
>>>>>>>>>
>>>>>>>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <
>>>>>>>>> jackagoodson@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Good idea, at the company I work at we discussed using Scala as
>>>>>>>>>> our primary language because technically it is slightly stronger than
>>>>>>>>>> python but ultimately chose python in the end as it’s easier for other devs
>>>>>>>>>> to be on boarded to our platform and future hiring for the team etc would
>>>>>>>>>> be easier
>>>>>>>>>>
>>>>>>>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <
>>>>>>>>>> gurwls223@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1 I like this idea too.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <
>>>>>>>>>>> afolting.dm@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to propose that we show Python code examples first
>>>>>>>>>>>> in the Spark documentation where we have multiple programming language
>>>>>>>>>>>> examples.
>>>>>>>>>>>> An example is on the Quick Start page:
>>>>>>>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>>>>>>>
>>>>>>>>>>>> I propose this change because Python has become more popular
>>>>>>>>>>>> than the other languages supported in Apache Spark. There are a lot more
>>>>>>>>>>>> users of Spark in Python than Scala today and Python attracts a broader set
>>>>>>>>>>>> of new users.
>>>>>>>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/
>>>>>>>>>>>> and
>>>>>>>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>> Also, this change aligns with Python already being the first
>>>>>>>>>>>> tab on our home page:
>>>>>>>>>>>> https://spark.apache.org/
>>>>>>>>>>>>
>>>>>>>>>>>> Anyone who wants to use another language can still just click
>>>>>>>>>>>> on the other tabs.
>>>>>>>>>>>>
>>>>>>>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets
>>>>>>>>>>>> Guide page as a first step:
>>>>>>>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>>>>>>>> proposal.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>>> Allan Folting
>>>>>>>>>>>>
>>>>>>>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Hyukjin Kwon <gu...@gmail.com>.
That sounds good to have that especially given that it will allow more
flexibility to the users.
But I think that's slightly orthogonal to this proposal since this proposal
is more about the default (before users take an action).
On Fri, 24 Feb 2023 at 15:35, Santosh Pingale <sa...@adyen.com>
wrote:
> Very interesting and user focused discussion, thanks for the proposal.
>
> Would it be better if we rather let users set the preference about the
> language they want to see first in the code examples? This preference can
> be easily stored on the browser side and used to decide ordering. This is
> inline with freedom users have with spark today.
>
>
> On Fri, Feb 24, 2023, 4:46 AM Allan Folting <af...@gmail.com> wrote:
>
>> I think this needs to be consistently done on all relevant pages and my
>> intent is to do that work in time for when it is first released.
>> I started with the "Spark SQL, DataFrames and Datasets Guide" page to
>> break it up into multiple, scoped PRs.
>> I should have made that clear before.
>>
>> I think it's a great idea to have an umbrella JIRA for this to outline
>> the full scope and track overall progress and I'm happy to create it.
>>
>> I can't speak on behalf of all Scala users of course, but I don't think
>> this change makes Scala appear as a 2nd class citizen, like I don't think
>> of Python as a 2nd class citizen because it is not first currently, but it
>> does recognize that Python is more broadly popular today.
>>
>> Thanks,
>> Allan
>>
>> On Thu, Feb 23, 2023 at 6:55 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Thank you all.
>>>
>>> Yes, attracting more Python users and being more Python user-friendly is
>>> always good.
>>>
>>> Basically, SPARK-42493 is proposing to introduce intentional
>>> inconsistency to Apache Spark documentation.
>>>
>>> The inconsistency from SPARK-42493 might give Python users the following
>>> questions first.
>>>
>>> - Why not RDD pages which are the heart of Apache Spark? Is Python not
>>> good in RDD?
>>> - Why not ML and Structured Streaming pages when DATA+AI Summit focuses
>>> on ML heavily?
>>>
>>> Also, more questions to the Scala users.
>>> - Is Scala language stepping down to the 2nd citizen language?
>>> - What about Scala 3?
>>>
>>> Of course, I understand SPARK-42493 has specific scopes
>>> (SQL/Dataset/Dataframe) and didn't mean anything like the above at all.
>>> However, if SPARK-42493 is emphasized as "the first step" to introduce
>>> that inconsistency, I'm wondering
>>> - What direction we are heading?
>>> - What is the next target scope?
>>> - When it will be achieved (or completed)?
>>> - Or, is the goal to be permanently inconsistent in terms of the
>>> documentation?
>>>
>>> It's unclear even in the documentation-only scope. If we are expecting
>>> more and more subtasks during Apache Spark 3.5 timeframe, shall we have an
>>> umbrella JIRA?
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Thu, Feb 23, 2023 at 6:15 PM Allan Folting <af...@gmail.com>
>>> wrote:
>>>
>>>> Thanks a lot for the questions and comments/feedback!
>>>>
>>>> To address your questions Dongjoon, I do not intend for these updates
>>>> to the documentation to be tied to the potential changes/suggestions you
>>>> ask about.
>>>>
>>>> In other words, this proposal is only about adjusting the documentation
>>>> to target the majority of people reading it - namely the large and growing
>>>> number of Python users - and new users in particular as they are often
>>>> already familiar with and have a preference for Python when evaluating or
>>>> starting to use Spark.
>>>>
>>>> While we may want to strengthen support for Python in other ways, I
>>>> think such efforts should be tracked separately from this.
>>>>
>>>> Allan
>>>>
>>>> On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> If this is not just flip flopping the document pages and involves
>>>>> other changes, then a proper impact analysis needs to be done to assess the
>>>>> efforts involved. Personally I don't think it really matters.
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> > 1. Does this suggestion imply Python API implementation will be the
>>>>>> new blocker in the future in terms of feature parity among languages? Until
>>>>>> now, Python API feature parity was one of the audit items because it's not
>>>>>> enforced. In other words, Scala and Java have been the full feature because
>>>>>> they are the underlying main developer languages while Python/R/SQL
>>>>>> environments were the nice-to-have.
>>>>>>
>>>>>> I think it wouldn't be treated as a blocker .. but I do believe we
>>>>>> have added all new features into the Python side for the last couple of
>>>>>> releases. So, I wouldn't worry about this at this moment - we have been
>>>>>> doing fine in terms of feature parity.
>>>>>>
>>>>>> > 2. Does this suggestion assume that the Python environment is
>>>>>> easier for users than Scala/Java always? Given that we support Python 3.8
>>>>>> to 3.11, the support matrix for Python library dependency is a problem for
>>>>>> the Apache Spark community to solve in order to claim that. As we say
>>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>>
>>>>>> Yes. In fact, regardless of this change, I do believe we should test
>>>>>> more versions, etc. At least scheduled jobs like we're doing JDK and Scala
>>>>>> versions.
>>>>>>
>>>>>>
>>>>>> FWIW, my take about this change is: people use Python and PySpark
>>>>>> more (according to the chart and stats provided) so let's put those
>>>>>> examples first :-).
>>>>>>
>>>>>>
>>>>>> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have two questions to clarify the scope and boundaries.
>>>>>>>
>>>>>>> 1. Does this suggestion imply Python API implementation will be the
>>>>>>> new blocker in the future in terms of feature parity among languages? Until
>>>>>>> now, Python API feature parity was one of the audit items because it's not
>>>>>>> enforced. In other words, Scala and Java have been the full feature because
>>>>>>> they are the underlying main developer languages while Python/R/SQL
>>>>>>> environments were the nice-to-have.
>>>>>>>
>>>>>>> 2. Does this suggestion assume that the Python environment is easier
>>>>>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>>>>>> the support matrix for Python library dependency is a problem for the
>>>>>>> Apache Spark community to solve in order to claim that. As we say
>>>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>>>
>>>>>>> Changing documentation is easy, but I hope we can give clear
>>>>>>> communication and direction in this effort because this is one of the most
>>>>>>> user-facing changes.
>>>>>>>
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <
>>>>>>> ruifengz@foxmail.com> wrote:
>>>>>>>
>>>>>>>> +1 LGTM
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> Ruifeng Zheng
>>>>>>>> ruifengz@foxmail.com
>>>>>>>>
>>>>>>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------ Original ------------------
>>>>>>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>>>>>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>>>>>>> *To:* "Allan Folting"<af...@gmail.com>;
>>>>>>>> *Cc:* "dev"<de...@spark.apache.org>;
>>>>>>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>>>>>>> documentation
>>>>>>>>
>>>>>>>> +1 Good idea!
>>>>>>>>
>>>>>>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <
>>>>>>>> jackagoodson@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Good idea, at the company I work at we discussed using Scala as
>>>>>>>>> our primary language because technically it is slightly stronger than
>>>>>>>>> python but ultimately chose python in the end as it’s easier for other devs
>>>>>>>>> to be on boarded to our platform and future hiring for the team etc would
>>>>>>>>> be easier
>>>>>>>>>
>>>>>>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 I like this idea too.
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <
>>>>>>>>>> afolting.dm@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I would like to propose that we show Python code examples first
>>>>>>>>>>> in the Spark documentation where we have multiple programming language
>>>>>>>>>>> examples.
>>>>>>>>>>> An example is on the Quick Start page:
>>>>>>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>>>>>>
>>>>>>>>>>> I propose this change because Python has become more popular
>>>>>>>>>>> than the other languages supported in Apache Spark. There are a lot more
>>>>>>>>>>> users of Spark in Python than Scala today and Python attracts a broader set
>>>>>>>>>>> of new users.
>>>>>>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/
>>>>>>>>>>> and
>>>>>>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> Also, this change aligns with Python already being the first tab
>>>>>>>>>>> on our home page:
>>>>>>>>>>> https://spark.apache.org/
>>>>>>>>>>>
>>>>>>>>>>> Anyone who wants to use another language can still just click on
>>>>>>>>>>> the other tabs.
>>>>>>>>>>>
>>>>>>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets
>>>>>>>>>>> Guide page as a first step:
>>>>>>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>>>>>>> proposal.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>> Allan Folting
>>>>>>>>>>>
>>>>>>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Santosh Pingale <sa...@adyen.com.INVALID>.
Very interesting and user focused discussion, thanks for the proposal.
Would it be better if we rather let users set the preference about the
language they want to see first in the code examples? This preference can
be easily stored on the browser side and used to decide ordering. This is
inline with freedom users have with spark today.
On Fri, Feb 24, 2023, 4:46 AM Allan Folting <af...@gmail.com> wrote:
> I think this needs to be consistently done on all relevant pages and my
> intent is to do that work in time for when it is first released.
> I started with the "Spark SQL, DataFrames and Datasets Guide" page to
> break it up into multiple, scoped PRs.
> I should have made that clear before.
>
> I think it's a great idea to have an umbrella JIRA for this to outline the
> full scope and track overall progress and I'm happy to create it.
>
> I can't speak on behalf of all Scala users of course, but I don't think
> this change makes Scala appear as a 2nd class citizen, like I don't think
> of Python as a 2nd class citizen because it is not first currently, but it
> does recognize that Python is more broadly popular today.
>
> Thanks,
> Allan
>
> On Thu, Feb 23, 2023 at 6:55 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Thank you all.
>>
>> Yes, attracting more Python users and being more Python user-friendly is
>> always good.
>>
>> Basically, SPARK-42493 is proposing to introduce intentional
>> inconsistency to Apache Spark documentation.
>>
>> The inconsistency from SPARK-42493 might give Python users the following
>> questions first.
>>
>> - Why not RDD pages which are the heart of Apache Spark? Is Python not
>> good in RDD?
>> - Why not ML and Structured Streaming pages when DATA+AI Summit focuses
>> on ML heavily?
>>
>> Also, more questions to the Scala users.
>> - Is Scala language stepping down to the 2nd citizen language?
>> - What about Scala 3?
>>
>> Of course, I understand SPARK-42493 has specific scopes
>> (SQL/Dataset/Dataframe) and didn't mean anything like the above at all.
>> However, if SPARK-42493 is emphasized as "the first step" to introduce
>> that inconsistency, I'm wondering
>> - What direction we are heading?
>> - What is the next target scope?
>> - When it will be achieved (or completed)?
>> - Or, is the goal to be permanently inconsistent in terms of the
>> documentation?
>>
>> It's unclear even in the documentation-only scope. If we are expecting
>> more and more subtasks during Apache Spark 3.5 timeframe, shall we have an
>> umbrella JIRA?
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Thu, Feb 23, 2023 at 6:15 PM Allan Folting <af...@gmail.com>
>> wrote:
>>
>>> Thanks a lot for the questions and comments/feedback!
>>>
>>> To address your questions Dongjoon, I do not intend for these updates to
>>> the documentation to be tied to the potential changes/suggestions you ask
>>> about.
>>>
>>> In other words, this proposal is only about adjusting the documentation
>>> to target the majority of people reading it - namely the large and growing
>>> number of Python users - and new users in particular as they are often
>>> already familiar with and have a preference for Python when evaluating or
>>> starting to use Spark.
>>>
>>> While we may want to strengthen support for Python in other ways, I
>>> think such efforts should be tracked separately from this.
>>>
>>> Allan
>>>
>>> On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> If this is not just flip flopping the document pages and involves other
>>>> changes, then a proper impact analysis needs to be done to assess the
>>>> efforts involved. Personally I don't think it really matters.
>>>>
>>>> HTH
>>>>
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com> wrote:
>>>>
>>>>> > 1. Does this suggestion imply Python API implementation will be the
>>>>> new blocker in the future in terms of feature parity among languages? Until
>>>>> now, Python API feature parity was one of the audit items because it's not
>>>>> enforced. In other words, Scala and Java have been the full feature because
>>>>> they are the underlying main developer languages while Python/R/SQL
>>>>> environments were the nice-to-have.
>>>>>
>>>>> I think it wouldn't be treated as a blocker .. but I do believe we
>>>>> have added all new features into the Python side for the last couple of
>>>>> releases. So, I wouldn't worry about this at this moment - we have been
>>>>> doing fine in terms of feature parity.
>>>>>
>>>>> > 2. Does this suggestion assume that the Python environment is easier
>>>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>>>> the support matrix for Python library dependency is a problem for the
>>>>> Apache Spark community to solve in order to claim that. As we say
>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>
>>>>> Yes. In fact, regardless of this change, I do believe we should test
>>>>> more versions, etc. At least scheduled jobs like we're doing JDK and Scala
>>>>> versions.
>>>>>
>>>>>
>>>>> FWIW, my take about this change is: people use Python and PySpark more
>>>>> (according to the chart and stats provided) so let's put those examples
>>>>> first :-).
>>>>>
>>>>>
>>>>> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I have two questions to clarify the scope and boundaries.
>>>>>>
>>>>>> 1. Does this suggestion imply Python API implementation will be the
>>>>>> new blocker in the future in terms of feature parity among languages? Until
>>>>>> now, Python API feature parity was one of the audit items because it's not
>>>>>> enforced. In other words, Scala and Java have been the full feature because
>>>>>> they are the underlying main developer languages while Python/R/SQL
>>>>>> environments were the nice-to-have.
>>>>>>
>>>>>> 2. Does this suggestion assume that the Python environment is easier
>>>>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>>>>> the support matrix for Python library dependency is a problem for the
>>>>>> Apache Spark community to solve in order to claim that. As we say
>>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>>
>>>>>> Changing documentation is easy, but I hope we can give clear
>>>>>> communication and direction in this effort because this is one of the most
>>>>>> user-facing changes.
>>>>>>
>>>>>> Dongjoon.
>>>>>>
>>>>>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <
>>>>>> ruifengz@foxmail.com> wrote:
>>>>>>
>>>>>>> +1 LGTM
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> Ruifeng Zheng
>>>>>>> ruifengz@foxmail.com
>>>>>>>
>>>>>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------ Original ------------------
>>>>>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>>>>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>>>>>> *To:* "Allan Folting"<af...@gmail.com>;
>>>>>>> *Cc:* "dev"<de...@spark.apache.org>;
>>>>>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>>>>>> documentation
>>>>>>>
>>>>>>> +1 Good idea!
>>>>>>>
>>>>>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Good idea, at the company I work at we discussed using Scala as our
>>>>>>>> primary language because technically it is slightly stronger than python
>>>>>>>> but ultimately chose python in the end as it’s easier for other devs to be
>>>>>>>> on boarded to our platform and future hiring for the team etc would be
>>>>>>>> easier
>>>>>>>>
>>>>>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 I like this idea too.
>>>>>>>>>
>>>>>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <
>>>>>>>>> afolting.dm@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I would like to propose that we show Python code examples first
>>>>>>>>>> in the Spark documentation where we have multiple programming language
>>>>>>>>>> examples.
>>>>>>>>>> An example is on the Quick Start page:
>>>>>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>>>>>
>>>>>>>>>> I propose this change because Python has become more popular than
>>>>>>>>>> the other languages supported in Apache Spark. There are a lot more users
>>>>>>>>>> of Spark in Python than Scala today and Python attracts a broader set of
>>>>>>>>>> new users.
>>>>>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/
>>>>>>>>>> and
>>>>>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>> Also, this change aligns with Python already being the first tab
>>>>>>>>>> on our home page:
>>>>>>>>>> https://spark.apache.org/
>>>>>>>>>>
>>>>>>>>>> Anyone who wants to use another language can still just click on
>>>>>>>>>> the other tabs.
>>>>>>>>>>
>>>>>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets
>>>>>>>>>> Guide page as a first step:
>>>>>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>>>>>> proposal.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks a lot,
>>>>>>>>>> Allan Folting
>>>>>>>>>>
>>>>>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Allan Folting <af...@gmail.com>.
I think this needs to be consistently done on all relevant pages and my
intent is to do that work in time for when it is first released.
I started with the "Spark SQL, DataFrames and Datasets Guide" page to break
it up into multiple, scoped PRs.
I should have made that clear before.
I think it's a great idea to have an umbrella JIRA for this to outline the
full scope and track overall progress and I'm happy to create it.
I can't speak on behalf of all Scala users of course, but I don't think
this change makes Scala appear as a 2nd class citizen, like I don't think
of Python as a 2nd class citizen because it is not first currently, but it
does recognize that Python is more broadly popular today.
Thanks,
Allan
On Thu, Feb 23, 2023 at 6:55 PM Dongjoon Hyun <do...@gmail.com>
wrote:
> Thank you all.
>
> Yes, attracting more Python users and being more Python user-friendly is
> always good.
>
> Basically, SPARK-42493 is proposing to introduce intentional inconsistency
> to Apache Spark documentation.
>
> The inconsistency from SPARK-42493 might give Python users the following
> questions first.
>
> - Why not RDD pages which are the heart of Apache Spark? Is Python not
> good in RDD?
> - Why not ML and Structured Streaming pages when DATA+AI Summit focuses on
> ML heavily?
>
> Also, more questions to the Scala users.
> - Is Scala language stepping down to the 2nd citizen language?
> - What about Scala 3?
>
> Of course, I understand SPARK-42493 has specific scopes
> (SQL/Dataset/Dataframe) and didn't mean anything like the above at all.
> However, if SPARK-42493 is emphasized as "the first step" to introduce
> that inconsistency, I'm wondering
> - What direction we are heading?
> - What is the next target scope?
> - When it will be achieved (or completed)?
> - Or, is the goal to be permanently inconsistent in terms of the
> documentation?
>
> It's unclear even in the documentation-only scope. If we are expecting
> more and more subtasks during Apache Spark 3.5 timeframe, shall we have an
> umbrella JIRA?
>
> Bests,
> Dongjoon.
>
>
> On Thu, Feb 23, 2023 at 6:15 PM Allan Folting <af...@gmail.com>
> wrote:
>
>> Thanks a lot for the questions and comments/feedback!
>>
>> To address your questions Dongjoon, I do not intend for these updates to
>> the documentation to be tied to the potential changes/suggestions you ask
>> about.
>>
>> In other words, this proposal is only about adjusting the documentation
>> to target the majority of people reading it - namely the large and growing
>> number of Python users - and new users in particular as they are often
>> already familiar with and have a preference for Python when evaluating or
>> starting to use Spark.
>>
>> While we may want to strengthen support for Python in other ways, I think
>> such efforts should be tracked separately from this.
>>
>> Allan
>>
>> On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> If this is not just flip flopping the document pages and involves other
>>> changes, then a proper impact analysis needs to be done to assess the
>>> efforts involved. Personally I don't think it really matters.
>>>
>>> HTH
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com> wrote:
>>>
>>>> > 1. Does this suggestion imply Python API implementation will be the
>>>> new blocker in the future in terms of feature parity among languages? Until
>>>> now, Python API feature parity was one of the audit items because it's not
>>>> enforced. In other words, Scala and Java have been the full feature because
>>>> they are the underlying main developer languages while Python/R/SQL
>>>> environments were the nice-to-have.
>>>>
>>>> I think it wouldn't be treated as a blocker .. but I do believe we have
>>>> added all new features into the Python side for the last couple of
>>>> releases. So, I wouldn't worry about this at this moment - we have been
>>>> doing fine in terms of feature parity.
>>>>
>>>> > 2. Does this suggestion assume that the Python environment is easier
>>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>>> the support matrix for Python library dependency is a problem for the
>>>> Apache Spark community to solve in order to claim that. As we say
>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>> historically and we have many `Pinned` python libraries issues.
>>>>
>>>> Yes. In fact, regardless of this change, I do believe we should test
>>>> more versions, etc. At least scheduled jobs like we're doing JDK and Scala
>>>> versions.
>>>>
>>>>
>>>> FWIW, my take about this change is: people use Python and PySpark more
>>>> (according to the chart and stats provided) so let's put those examples
>>>> first :-).
>>>>
>>>>
>>>> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
>>>> wrote:
>>>>
>>>>> I have two questions to clarify the scope and boundaries.
>>>>>
>>>>> 1. Does this suggestion imply Python API implementation will be the
>>>>> new blocker in the future in terms of feature parity among languages? Until
>>>>> now, Python API feature parity was one of the audit items because it's not
>>>>> enforced. In other words, Scala and Java have been the full feature because
>>>>> they are the underlying main developer languages while Python/R/SQL
>>>>> environments were the nice-to-have.
>>>>>
>>>>> 2. Does this suggestion assume that the Python environment is easier
>>>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>>>> the support matrix for Python library dependency is a problem for the
>>>>> Apache Spark community to solve in order to claim that. As we say
>>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>>> historically and we have many `Pinned` python libraries issues.
>>>>>
>>>>> Changing documentation is easy, but I hope we can give clear
>>>>> communication and direction in this effort because this is one of the most
>>>>> user-facing changes.
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ru...@foxmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1 LGTM
>>>>>>
>>>>>> ------------------------------
>>>>>> Ruifeng Zheng
>>>>>> ruifengz@foxmail.com
>>>>>>
>>>>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------ Original ------------------
>>>>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>>>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>>>>> *To:* "Allan Folting"<af...@gmail.com>;
>>>>>> *Cc:* "dev"<de...@spark.apache.org>;
>>>>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>>>>> documentation
>>>>>>
>>>>>> +1 Good idea!
>>>>>>
>>>>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Good idea, at the company I work at we discussed using Scala as our
>>>>>>> primary language because technically it is slightly stronger than python
>>>>>>> but ultimately chose python in the end as it’s easier for other devs to be
>>>>>>> on boarded to our platform and future hiring for the team etc would be
>>>>>>> easier
>>>>>>>
>>>>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 I like this idea too.
>>>>>>>>
>>>>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <
>>>>>>>> afolting.dm@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I would like to propose that we show Python code examples first in
>>>>>>>>> the Spark documentation where we have multiple programming language
>>>>>>>>> examples.
>>>>>>>>> An example is on the Quick Start page:
>>>>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>>>>
>>>>>>>>> I propose this change because Python has become more popular than
>>>>>>>>> the other languages supported in Apache Spark. There are a lot more users
>>>>>>>>> of Spark in Python than Scala today and Python attracts a broader set of
>>>>>>>>> new users.
>>>>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> Also, this change aligns with Python already being the first tab
>>>>>>>>> on our home page:
>>>>>>>>> https://spark.apache.org/
>>>>>>>>>
>>>>>>>>> Anyone who wants to use another language can still just click on
>>>>>>>>> the other tabs.
>>>>>>>>>
>>>>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets
>>>>>>>>> Guide page as a first step:
>>>>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>>>>> proposal.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks a lot,
>>>>>>>>> Allan Folting
>>>>>>>>>
>>>>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you all.
Yes, attracting more Python users and being more Python user-friendly is
always good.
Basically, SPARK-42493 is proposing to introduce intentional inconsistency
to Apache Spark documentation.
The inconsistency from SPARK-42493 might give Python users the following
questions first.
- Why not RDD pages which are the heart of Apache Spark? Is Python not good
in RDD?
- Why not ML and Structured Streaming pages when DATA+AI Summit focuses on
ML heavily?
Also, more questions to the Scala users.
- Is Scala language stepping down to the 2nd citizen language?
- What about Scala 3?
Of course, I understand SPARK-42493 has specific scopes
(SQL/Dataset/Dataframe) and didn't mean anything like the above at all.
However, if SPARK-42493 is emphasized as "the first step" to introduce that
inconsistency, I'm wondering
- What direction we are heading?
- What is the next target scope?
- When it will be achieved (or completed)?
- Or, is the goal to be permanently inconsistent in terms of the
documentation?
It's unclear even in the documentation-only scope. If we are expecting more
and more subtasks during Apache Spark 3.5 timeframe, shall we have an
umbrella JIRA?
Bests,
Dongjoon.
On Thu, Feb 23, 2023 at 6:15 PM Allan Folting <af...@gmail.com> wrote:
> Thanks a lot for the questions and comments/feedback!
>
> To address your questions Dongjoon, I do not intend for these updates to
> the documentation to be tied to the potential changes/suggestions you ask
> about.
>
> In other words, this proposal is only about adjusting the documentation to
> target the majority of people reading it - namely the large and growing
> number of Python users - and new users in particular as they are often
> already familiar with and have a preference for Python when evaluating or
> starting to use Spark.
>
> While we may want to strengthen support for Python in other ways, I think
> such efforts should be tracked separately from this.
>
> Allan
>
> On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> If this is not just flip flopping the document pages and involves other
>> changes, then a proper impact analysis needs to be done to assess the
>> efforts involved. Personally I don't think it really matters.
>>
>> HTH
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> > 1. Does this suggestion imply Python API implementation will be the
>>> new blocker in the future in terms of feature parity among languages? Until
>>> now, Python API feature parity was one of the audit items because it's not
>>> enforced. In other words, Scala and Java have been the full feature because
>>> they are the underlying main developer languages while Python/R/SQL
>>> environments were the nice-to-have.
>>>
>>> I think it wouldn't be treated as a blocker .. but I do believe we have
>>> added all new features into the Python side for the last couple of
>>> releases. So, I wouldn't worry about this at this moment - we have been
>>> doing fine in terms of feature parity.
>>>
>>> > 2. Does this suggestion assume that the Python environment is easier
>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>> the support matrix for Python library dependency is a problem for the
>>> Apache Spark community to solve in order to claim that. As we say
>>> at SPARK-41454, Python language also introduces breaking changes to us
>>> historically and we have many `Pinned` python libraries issues.
>>>
>>> Yes. In fact, regardless of this change, I do believe we should test
>>> more versions, etc. At least scheduled jobs like we're doing JDK and Scala
>>> versions.
>>>
>>>
>>> FWIW, my take about this change is: people use Python and PySpark more
>>> (according to the chart and stats provided) so let's put those examples
>>> first :-).
>>>
>>>
>>> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> I have two questions to clarify the scope and boundaries.
>>>>
>>>> 1. Does this suggestion imply Python API implementation will be the new
>>>> blocker in the future in terms of feature parity among languages? Until
>>>> now, Python API feature parity was one of the audit items because it's not
>>>> enforced. In other words, Scala and Java have been the full feature because
>>>> they are the underlying main developer languages while Python/R/SQL
>>>> environments were the nice-to-have.
>>>>
>>>> 2. Does this suggestion assume that the Python environment is easier
>>>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>>>> the support matrix for Python library dependency is a problem for the
>>>> Apache Spark community to solve in order to claim that. As we say
>>>> at SPARK-41454, Python language also introduces breaking changes to us
>>>> historically and we have many `Pinned` python libraries issues.
>>>>
>>>> Changing documentation is easy, but I hope we can give clear
>>>> communication and direction in this effort because this is one of the most
>>>> user-facing changes.
>>>>
>>>> Dongjoon.
>>>>
>>>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ru...@foxmail.com>
>>>> wrote:
>>>>
>>>>> +1 LGTM
>>>>>
>>>>> ------------------------------
>>>>> Ruifeng Zheng
>>>>> ruifengz@foxmail.com
>>>>>
>>>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>>>
>>>>>
>>>>>
>>>>> ------------------ Original ------------------
>>>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>>>> *To:* "Allan Folting"<af...@gmail.com>;
>>>>> *Cc:* "dev"<de...@spark.apache.org>;
>>>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>>>> documentation
>>>>>
>>>>> +1 Good idea!
>>>>>
>>>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Good idea, at the company I work at we discussed using Scala as our
>>>>>> primary language because technically it is slightly stronger than python
>>>>>> but ultimately chose python in the end as it’s easier for other devs to be
>>>>>> on boarded to our platform and future hiring for the team etc would be
>>>>>> easier
>>>>>>
>>>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 I like this idea too.
>>>>>>>
>>>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I would like to propose that we show Python code examples first in
>>>>>>>> the Spark documentation where we have multiple programming language
>>>>>>>> examples.
>>>>>>>> An example is on the Quick Start page:
>>>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>>>
>>>>>>>> I propose this change because Python has become more popular than
>>>>>>>> the other languages supported in Apache Spark. There are a lot more users
>>>>>>>> of Spark in Python than Scala today and Python attracts a broader set of
>>>>>>>> new users.
>>>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>>>> .
>>>>>>>>
>>>>>>>> Also, this change aligns with Python already being the first tab on
>>>>>>>> our home page:
>>>>>>>> https://spark.apache.org/
>>>>>>>>
>>>>>>>> Anyone who wants to use another language can still just click on
>>>>>>>> the other tabs.
>>>>>>>>
>>>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets
>>>>>>>> Guide page as a first step:
>>>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>>>
>>>>>>>>
>>>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>>>> proposal.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks a lot,
>>>>>>>> Allan Folting
>>>>>>>>
>>>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Allan Folting <af...@gmail.com>.
Thanks a lot for the questions and comments/feedback!
To address your questions Dongjoon, I do not intend for these updates to
the documentation to be tied to the potential changes/suggestions you ask
about.
In other words, this proposal is only about adjusting the documentation to
target the majority of people reading it - namely the large and growing
number of Python users - and new users in particular as they are often
already familiar with and have a preference for Python when evaluating or
starting to use Spark.
While we may want to strengthen support for Python in other ways, I think
such efforts should be tracked separately from this.
Allan
On Thu, Feb 23, 2023 at 1:44 AM Mich Talebzadeh <mi...@gmail.com>
wrote:
> If this is not just flip flopping the document pages and involves other
> changes, then a proper impact analysis needs to be done to assess the
> efforts involved. Personally I don't think it really matters.
>
> HTH
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> > 1. Does this suggestion imply Python API implementation will be the new
>> blocker in the future in terms of feature parity among languages? Until
>> now, Python API feature parity was one of the audit items because it's not
>> enforced. In other words, Scala and Java have been the full feature because
>> they are the underlying main developer languages while Python/R/SQL
>> environments were the nice-to-have.
>>
>> I think it wouldn't be treated as a blocker .. but I do believe we have
>> added all new features into the Python side for the last couple of
>> releases. So, I wouldn't worry about this at this moment - we have been
>> doing fine in terms of feature parity.
>>
>> > 2. Does this suggestion assume that the Python environment is easier
>> for users than Scala/Java always? Given that we support Python 3.8 to 3.11,
>> the support matrix for Python library dependency is a problem for the
>> Apache Spark community to solve in order to claim that. As we say
>> at SPARK-41454, Python language also introduces breaking changes to us
>> historically and we have many `Pinned` python libraries issues.
>>
>> Yes. In fact, regardless of this change, I do believe we should test more
>> versions, etc. At least scheduled jobs like we're doing JDK and Scala
>> versions.
>>
>>
>> FWIW, my take about this change is: people use Python and PySpark more
>> (according to the chart and stats provided) so let's put those examples
>> first :-).
>>
>>
>> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> I have two questions to clarify the scope and boundaries.
>>>
>>> 1. Does this suggestion imply Python API implementation will be the new
>>> blocker in the future in terms of feature parity among languages? Until
>>> now, Python API feature parity was one of the audit items because it's not
>>> enforced. In other words, Scala and Java have been the full feature because
>>> they are the underlying main developer languages while Python/R/SQL
>>> environments were the nice-to-have.
>>>
>>> 2. Does this suggestion assume that the Python environment is easier for
>>> users than Scala/Java always? Given that we support Python 3.8 to 3.11, the
>>> support matrix for Python library dependency is a problem for the Apache
>>> Spark community to solve in order to claim that. As we say at SPARK-41454,
>>> Python language also introduces breaking changes to us historically and we
>>> have many `Pinned` python libraries issues.
>>>
>>> Changing documentation is easy, but I hope we can give clear
>>> communication and direction in this effort because this is one of the most
>>> user-facing changes.
>>>
>>> Dongjoon.
>>>
>>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ru...@foxmail.com>
>>> wrote:
>>>
>>>> +1 LGTM
>>>>
>>>> ------------------------------
>>>> Ruifeng Zheng
>>>> ruifengz@foxmail.com
>>>>
>>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>>
>>>>
>>>>
>>>> ------------------ Original ------------------
>>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>>> *To:* "Allan Folting"<af...@gmail.com>;
>>>> *Cc:* "dev"<de...@spark.apache.org>;
>>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>>> documentation
>>>>
>>>> +1 Good idea!
>>>>
>>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
>>>> wrote:
>>>>
>>>>> Good idea, at the company I work at we discussed using Scala as our
>>>>> primary language because technically it is slightly stronger than python
>>>>> but ultimately chose python in the end as it’s easier for other devs to be
>>>>> on boarded to our platform and future hiring for the team etc would be
>>>>> easier
>>>>>
>>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1 I like this idea too.
>>>>>>
>>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I would like to propose that we show Python code examples first in
>>>>>>> the Spark documentation where we have multiple programming language
>>>>>>> examples.
>>>>>>> An example is on the Quick Start page:
>>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>>
>>>>>>> I propose this change because Python has become more popular than
>>>>>>> the other languages supported in Apache Spark. There are a lot more users
>>>>>>> of Spark in Python than Scala today and Python attracts a broader set of
>>>>>>> new users.
>>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>>> .
>>>>>>>
>>>>>>> Also, this change aligns with Python already being the first tab on
>>>>>>> our home page:
>>>>>>> https://spark.apache.org/
>>>>>>>
>>>>>>> Anyone who wants to use another language can still just click on the
>>>>>>> other tabs.
>>>>>>>
>>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets
>>>>>>> Guide page as a first step:
>>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>>
>>>>>>>
>>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>>> proposal.
>>>>>>>
>>>>>>>
>>>>>>> Thanks a lot,
>>>>>>> Allan Folting
>>>>>>>
>>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Mich Talebzadeh <mi...@gmail.com>.
If this is not just flip flopping the document pages and involves other
changes, then a proper impact analysis needs to be done to assess the
efforts involved. Personally I don't think it really matters.
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 23 Feb 2023 at 01:40, Hyukjin Kwon <gu...@gmail.com> wrote:
> > 1. Does this suggestion imply Python API implementation will be the new
> blocker in the future in terms of feature parity among languages? Until
> now, Python API feature parity was one of the audit items because it's not
> enforced. In other words, Scala and Java have been the full feature because
> they are the underlying main developer languages while Python/R/SQL
> environments were the nice-to-have.
>
> I think it wouldn't be treated as a blocker .. but I do believe we have
> added all new features into the Python side for the last couple of
> releases. So, I wouldn't worry about this at this moment - we have been
> doing fine in terms of feature parity.
>
> > 2. Does this suggestion assume that the Python environment is easier for
> users than Scala/Java always? Given that we support Python 3.8 to 3.11, the
> support matrix for Python library dependency is a problem for the Apache
> Spark community to solve in order to claim that. As we say at SPARK-41454,
> Python language also introduces breaking changes to us historically and we
> have many `Pinned` python libraries issues.
>
> Yes. In fact, regardless of this change, I do believe we should test more
> versions, etc. At least scheduled jobs like we're doing JDK and Scala
> versions.
>
>
> FWIW, my take about this change is: people use Python and PySpark more
> (according to the chart and stats provided) so let's put those examples
> first :-).
>
>
> On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> I have two questions to clarify the scope and boundaries.
>>
>> 1. Does this suggestion imply Python API implementation will be the new
>> blocker in the future in terms of feature parity among languages? Until
>> now, Python API feature parity was one of the audit items because it's not
>> enforced. In other words, Scala and Java have been the full feature because
>> they are the underlying main developer languages while Python/R/SQL
>> environments were the nice-to-have.
>>
>> 2. Does this suggestion assume that the Python environment is easier for
>> users than Scala/Java always? Given that we support Python 3.8 to 3.11, the
>> support matrix for Python library dependency is a problem for the Apache
>> Spark community to solve in order to claim that. As we say at SPARK-41454,
>> Python language also introduces breaking changes to us historically and we
>> have many `Pinned` python libraries issues.
>>
>> Changing documentation is easy, but I hope we can give clear
>> communication and direction in this effort because this is one of the most
>> user-facing changes.
>>
>> Dongjoon.
>>
>> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ru...@foxmail.com>
>> wrote:
>>
>>> +1 LGTM
>>>
>>> ------------------------------
>>> Ruifeng Zheng
>>> ruifengz@foxmail.com
>>>
>>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>>
>>>
>>>
>>> ------------------ Original ------------------
>>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>>> *Date:* Thu, Feb 23, 2023 09:17 AM
>>> *To:* "Allan Folting"<af...@gmail.com>;
>>> *Cc:* "dev"<de...@spark.apache.org>;
>>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>>> documentation
>>>
>>> +1 Good idea!
>>>
>>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
>>> wrote:
>>>
>>>> Good idea, at the company I work at we discussed using Scala as our
>>>> primary language because technically it is slightly stronger than python
>>>> but ultimately chose python in the end as it’s easier for other devs to be
>>>> on boarded to our platform and future hiring for the team etc would be
>>>> easier
>>>>
>>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 I like this idea too.
>>>>>
>>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I would like to propose that we show Python code examples first in
>>>>>> the Spark documentation where we have multiple programming language
>>>>>> examples.
>>>>>> An example is on the Quick Start page:
>>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>>
>>>>>> I propose this change because Python has become more popular than the
>>>>>> other languages supported in Apache Spark. There are a lot more users of
>>>>>> Spark in Python than Scala today and Python attracts a broader set of new
>>>>>> users.
>>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>>> .
>>>>>>
>>>>>> Also, this change aligns with Python already being the first tab on
>>>>>> our home page:
>>>>>> https://spark.apache.org/
>>>>>>
>>>>>> Anyone who wants to use another language can still just click on the
>>>>>> other tabs.
>>>>>>
>>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>>>>>> page as a first step:
>>>>>> https://github.com/apache/spark/pull/40087
>>>>>>
>>>>>>
>>>>>> I would appreciate it if you could share your thoughts on this
>>>>>> proposal.
>>>>>>
>>>>>>
>>>>>> Thanks a lot,
>>>>>> Allan Folting
>>>>>>
>>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Hyukjin Kwon <gu...@gmail.com>.
> 1. Does this suggestion imply Python API implementation will be the new
blocker in the future in terms of feature parity among languages? Until
now, Python API feature parity was one of the audit items because it's not
enforced. In other words, Scala and Java have been the full feature because
they are the underlying main developer languages while Python/R/SQL
environments were the nice-to-have.
I think it wouldn't be treated as a blocker .. but I do believe we have
added all new features into the Python side for the last couple of
releases. So, I wouldn't worry about this at this moment - we have been
doing fine in terms of feature parity.
> 2. Does this suggestion assume that the Python environment is easier for
users than Scala/Java always? Given that we support Python 3.8 to 3.11, the
support matrix for Python library dependency is a problem for the Apache
Spark community to solve in order to claim that. As we say at SPARK-41454,
Python language also introduces breaking changes to us historically and we
have many `Pinned` python libraries issues.
Yes. In fact, regardless of this change, I do believe we should test more
versions, etc. At least scheduled jobs like we're doing JDK and Scala
versions.
FWIW, my take about this change is: people use Python and PySpark more
(according to the chart and stats provided) so let's put those examples
first :-).
On Thu, 23 Feb 2023 at 10:27, Dongjoon Hyun <do...@gmail.com> wrote:
> I have two questions to clarify the scope and boundaries.
>
> 1. Does this suggestion imply Python API implementation will be the new
> blocker in the future in terms of feature parity among languages? Until
> now, Python API feature parity was one of the audit items because it's not
> enforced. In other words, Scala and Java have been the full feature because
> they are the underlying main developer languages while Python/R/SQL
> environments were the nice-to-have.
>
> 2. Does this suggestion assume that the Python environment is easier for
> users than Scala/Java always? Given that we support Python 3.8 to 3.11, the
> support matrix for Python library dependency is a problem for the Apache
> Spark community to solve in order to claim that. As we say at SPARK-41454,
> Python language also introduces breaking changes to us historically and we
> have many `Pinned` python libraries issues.
>
> Changing documentation is easy, but I hope we can give clear
> communication and direction in this effort because this is one of the most
> user-facing changes.
>
> Dongjoon.
>
> On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ru...@foxmail.com>
> wrote:
>
>> +1 LGTM
>>
>> ------------------------------
>> Ruifeng Zheng
>> ruifengz@foxmail.com
>>
>> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>>
>>
>>
>> ------------------ Original ------------------
>> *From:* "Xinrong Meng" <xi...@gmail.com>;
>> *Date:* Thu, Feb 23, 2023 09:17 AM
>> *To:* "Allan Folting"<af...@gmail.com>;
>> *Cc:* "dev"<de...@spark.apache.org>;
>> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
>> documentation
>>
>> +1 Good idea!
>>
>> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
>> wrote:
>>
>>> Good idea, at the company I work at we discussed using Scala as our
>>> primary language because technically it is slightly stronger than python
>>> but ultimately chose python in the end as it’s easier for other devs to be
>>> on boarded to our platform and future hiring for the team etc would be
>>> easier
>>>
>>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> +1 I like this idea too.
>>>>
>>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I would like to propose that we show Python code examples first in the
>>>>> Spark documentation where we have multiple programming language examples.
>>>>> An example is on the Quick Start page:
>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>>
>>>>> I propose this change because Python has become more popular than the
>>>>> other languages supported in Apache Spark. There are a lot more users of
>>>>> Spark in Python than Scala today and Python attracts a broader set of new
>>>>> users.
>>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>>> .
>>>>>
>>>>> Also, this change aligns with Python already being the first tab on
>>>>> our home page:
>>>>> https://spark.apache.org/
>>>>>
>>>>> Anyone who wants to use another language can still just click on the
>>>>> other tabs.
>>>>>
>>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>>>>> page as a first step:
>>>>> https://github.com/apache/spark/pull/40087
>>>>>
>>>>>
>>>>> I would appreciate it if you could share your thoughts on this
>>>>> proposal.
>>>>>
>>>>>
>>>>> Thanks a lot,
>>>>> Allan Folting
>>>>>
>>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Dongjoon Hyun <do...@gmail.com>.
I have two questions to clarify the scope and boundaries.
1. Does this suggestion imply Python API implementation will be the new
blocker in the future in terms of feature parity among languages? Until
now, Python API feature parity was one of the audit items because it's not
enforced. In other words, Scala and Java have been the full feature because
they are the underlying main developer languages while Python/R/SQL
environments were the nice-to-have.
2. Does this suggestion assume that the Python environment is easier for
users than Scala/Java always? Given that we support Python 3.8 to 3.11, the
support matrix for Python library dependency is a problem for the Apache
Spark community to solve in order to claim that. As we say at SPARK-41454,
Python language also introduces breaking changes to us historically and we
have many `Pinned` python libraries issues.
Changing documentation is easy, but I hope we can give clear
communication and direction in this effort because this is one of the most
user-facing changes.
Dongjoon.
On Wed, Feb 22, 2023 at 5:26 PM 416161436@qq.com <ru...@foxmail.com>
wrote:
> +1 LGTM
>
> ------------------------------
> Ruifeng Zheng
> ruifengz@foxmail.com
>
> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=Ruifeng+Zheng&icon=https%3A%2F%2Fres.mail.qq.com%2Fzh_CN%2Fhtmledition%2Fimages%2Frss%2Fmale.gif%3Frand%3D1617349242&mail=ruifengz%40foxmail.com&code=>
>
>
>
> ------------------ Original ------------------
> *From:* "Xinrong Meng" <xi...@gmail.com>;
> *Date:* Thu, Feb 23, 2023 09:17 AM
> *To:* "Allan Folting"<af...@gmail.com>;
> *Cc:* "dev"<de...@spark.apache.org>;
> *Subject:* Re: [DISCUSS] Show Python code examples first in Spark
> documentation
>
> +1 Good idea!
>
> On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com>
> wrote:
>
>> Good idea, at the company I work at we discussed using Scala as our
>> primary language because technically it is slightly stronger than python
>> but ultimately chose python in the end as it’s easier for other devs to be
>> on boarded to our platform and future hiring for the team etc would be
>> easier
>>
>> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>>
>>> +1 I like this idea too.
>>>
>>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I would like to propose that we show Python code examples first in the
>>>> Spark documentation where we have multiple programming language examples.
>>>> An example is on the Quick Start page:
>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>
>>>> I propose this change because Python has become more popular than the
>>>> other languages supported in Apache Spark. There are a lot more users of
>>>> Spark in Python than Scala today and Python attracts a broader set of new
>>>> users.
>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>> .
>>>>
>>>> Also, this change aligns with Python already being the first tab on our
>>>> home page:
>>>> https://spark.apache.org/
>>>>
>>>> Anyone who wants to use another language can still just click on the
>>>> other tabs.
>>>>
>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>>>> page as a first step:
>>>> https://github.com/apache/spark/pull/40087
>>>>
>>>>
>>>> I would appreciate it if you could share your thoughts on this proposal.
>>>>
>>>>
>>>> Thanks a lot,
>>>> Allan Folting
>>>>
>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by "416161436@qq.com" <ru...@foxmail.com>.
+1 LGTM
Ruifeng Zheng
ruifengz@foxmail.com
------------------ Original ------------------
From: "Xinrong Meng" <xinrong.apache@gmail.com>;
Date: Thu, Feb 23, 2023 09:17 AM
To: "Allan Folting"<afolting.dm@gmail.com>;
Cc: "dev"<dev@spark.apache.org>;
Subject: Re: [DISCUSS] Show Python code examples first in Spark documentation
+1 Good idea!
On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <jackagoodson@gmail.com> wrote:
Good idea, at the company I work at we discussed using Scala as our primary language because technically it is slightly stronger than python but ultimately chose python in the end as it’s easier for other devs to be on boarded to our platform and future hiring for the team etc would be easier
On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gurwls223@gmail.com> wrote:
+1 I like this idea too.
On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <afolting.dm@gmail.com> wrote:
Hi all,
I would like to propose that we show Python code examples first in the Spark documentation where we have multiple programming language examples.
An example is on the Quick Start page:
https://spark.apache.org/docs/latest/quick-start.html
I propose this change because Python has become more popular than the other languages supported in Apache Spark. There are a lot more users of Spark in Python than Scala today and Python attracts a broader set of new users.
For Python usage data, see https://www.tiobe.com/tiobe-index/ and https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
Also, this change aligns with Python already being the first tab on our home page:
https://spark.apache.org/
Anyone who wants to use another language can still just click on the other tabs.
I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page as a first step:
https://github.com/apache/spark/pull/40087
I would appreciate it if you could share your thoughts on this proposal.
Thanks a lot,
Allan Folting
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Xinrong Meng <xi...@gmail.com>.
+1 Good idea!
On Thu, Feb 23, 2023 at 7:41 AM Jack Goodson <ja...@gmail.com> wrote:
> Good idea, at the company I work at we discussed using Scala as our
> primary language because technically it is slightly stronger than python
> but ultimately chose python in the end as it’s easier for other devs to be
> on boarded to our platform and future hiring for the team etc would be
> easier
>
> On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> +1 I like this idea too.
>>
>> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I would like to propose that we show Python code examples first in the
>>> Spark documentation where we have multiple programming language examples.
>>> An example is on the Quick Start page:
>>> https://spark.apache.org/docs/latest/quick-start.html
>>>
>>> I propose this change because Python has become more popular than the
>>> other languages supported in Apache Spark. There are a lot more users of
>>> Spark in Python than Scala today and Python attracts a broader set of new
>>> users.
>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>> .
>>>
>>> Also, this change aligns with Python already being the first tab on our
>>> home page:
>>> https://spark.apache.org/
>>>
>>> Anyone who wants to use another language can still just click on the
>>> other tabs.
>>>
>>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>>> page as a first step:
>>> https://github.com/apache/spark/pull/40087
>>>
>>>
>>> I would appreciate it if you could share your thoughts on this proposal.
>>>
>>>
>>> Thanks a lot,
>>> Allan Folting
>>>
>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Jack Goodson <ja...@gmail.com>.
Good idea, at the company I work at we discussed using Scala as our primary
language because technically it is slightly stronger than python but
ultimately chose python in the end as it’s easier for other devs to be on
boarded to our platform and future hiring for the team etc would be easier
On Thu, 23 Feb 2023 at 12:20 PM, Hyukjin Kwon <gu...@gmail.com> wrote:
> +1 I like this idea too.
>
> On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I would like to propose that we show Python code examples first in the
>> Spark documentation where we have multiple programming language examples.
>> An example is on the Quick Start page:
>> https://spark.apache.org/docs/latest/quick-start.html
>>
>> I propose this change because Python has become more popular than the
>> other languages supported in Apache Spark. There are a lot more users of
>> Spark in Python than Scala today and Python attracts a broader set of new
>> users.
>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
>>
>> Also, this change aligns with Python already being the first tab on our
>> home page:
>> https://spark.apache.org/
>>
>> Anyone who wants to use another language can still just click on the
>> other tabs.
>>
>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>> page as a first step:
>> https://github.com/apache/spark/pull/40087
>>
>>
>> I would appreciate it if you could share your thoughts on this proposal.
>>
>>
>> Thanks a lot,
>> Allan Folting
>>
>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Hyukjin Kwon <gu...@gmail.com>.
+1 I like this idea too.
On Thu, Feb 23, 2023 at 6:00 AM Allan Folting <af...@gmail.com> wrote:
> Hi all,
>
> I would like to propose that we show Python code examples first in the
> Spark documentation where we have multiple programming language examples.
> An example is on the Quick Start page:
> https://spark.apache.org/docs/latest/quick-start.html
>
> I propose this change because Python has become more popular than the
> other languages supported in Apache Spark. There are a lot more users of
> Spark in Python than Scala today and Python attracts a broader set of new
> users.
> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
>
> Also, this change aligns with Python already being the first tab on our
> home page:
> https://spark.apache.org/
>
> Anyone who wants to use another language can still just click on the other
> tabs.
>
> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page
> as a first step:
> https://github.com/apache/spark/pull/40087
>
>
> I would appreciate it if you could share your thoughts on this proposal.
>
>
> Thanks a lot,
> Allan Folting
>
Fwd: 自动回复: Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
Can someone disable the below login from spark forums please?
Sounds like someone left this email and we are receiving a spam type
message anytime we respond.
thanks
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
---------- Forwarded message ---------
From: xyqiao <xy...@vip.qq.com>
Date: Sun, 26 Feb 2023 at 22:42
Subject: 自动回复: Re: [DISCUSS] Show Python code examples first in Spark
documentation
To: Mich Talebzadeh <mi...@gmail.com>
这是来自QQ邮箱的假期自动回复邮件。
您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。
Fwd: 自动回复: Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
Can someone disable the below login from spark forums please?
Sounds like someone left this email and we are receiving a spam type
message anytime we respond.
thanks
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
---------- Forwarded message ---------
From: xyqiao <xy...@vip.qq.com>
Date: Sun, 26 Feb 2023 at 22:42
Subject: 自动回复: Re: [DISCUSS] Show Python code examples first in Spark
documentation
To: Mich Talebzadeh <mi...@gmail.com>
这是来自QQ邮箱的假期自动回复邮件。
您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Hyukjin Kwon <gu...@gmail.com>.
This is already getting too long to follow the original topic. Would be
great if we can separate others to a different thread.
On Mon, Feb 27, 2023 at 7:41 AM Mich Talebzadeh <mi...@gmail.com>
wrote:
> if we are going to do it, we might as well do it all. it is more cost
> effective so to speak.
>
>
> HTH
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 26 Feb 2023 at 21:48, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Probably it's worthwhile discussing the order for others but I would keep
>> it separate from this thread to focus on Python as the default since that
>> can be done as an incremental improvement.
>>
>>
>> On Mon, Feb 27, 2023 at 3:36 AM Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>>
>>> To me as I stated before this is a tangential issue. However, if it has
>>> to be done, we ought to work out how much effort is needed). Besides, how
>>> about others?
>>>
>>>
>>> [image: image.png]
>>>
>>> what next in the order?
>>>
>>>
>>>
>>> 1. Python
>>> 2. Scala
>>> 3. SQL
>>> 4. Java
>>> 5. R
>>>
>>> SQL is still very important and with Spark becoming more relevant to ETL
>>> plus lesser extend to DS, I see the above order is fair
>>>
>>> HTH
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Wed, 22 Feb 2023 at 21:00, Allan Folting <af...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I would like to propose that we show Python code examples first in the
>>>> Spark documentation where we have multiple programming language examples.
>>>> An example is on the Quick Start page:
>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>
>>>> I propose this change because Python has become more popular than the
>>>> other languages supported in Apache Spark. There are a lot more users of
>>>> Spark in Python than Scala today and Python attracts a broader set of new
>>>> users.
>>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>>> .
>>>>
>>>> Also, this change aligns with Python already being the first tab on our
>>>> home page:
>>>> https://spark.apache.org/
>>>>
>>>> Anyone who wants to use another language can still just click on the
>>>> other tabs.
>>>>
>>>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>>>> page as a first step:
>>>> https://github.com/apache/spark/pull/40087
>>>>
>>>>
>>>> I would appreciate it if you could share your thoughts on this proposal.
>>>>
>>>>
>>>> Thanks a lot,
>>>> Allan Folting
>>>>
>>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Mich Talebzadeh <mi...@gmail.com>.
if we are going to do it, we might as well do it all. it is more cost
effective so to speak.
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 26 Feb 2023 at 21:48, Hyukjin Kwon <gu...@gmail.com> wrote:
> Probably it's worthwhile discussing the order for others but I would keep
> it separate from this thread to focus on Python as the default since that
> can be done as an incremental improvement.
>
>
> On Mon, Feb 27, 2023 at 3:36 AM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>>
>> To me as I stated before this is a tangential issue. However, if it has
>> to be done, we ought to work out how much effort is needed). Besides, how
>> about others?
>>
>>
>> [image: image.png]
>>
>> what next in the order?
>>
>>
>>
>> 1. Python
>> 2. Scala
>> 3. SQL
>> 4. Java
>> 5. R
>>
>> SQL is still very important and with Spark becoming more relevant to ETL
>> plus lesser extend to DS, I see the above order is fair
>>
>> HTH
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Wed, 22 Feb 2023 at 21:00, Allan Folting <af...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I would like to propose that we show Python code examples first in the
>>> Spark documentation where we have multiple programming language examples.
>>> An example is on the Quick Start page:
>>> https://spark.apache.org/docs/latest/quick-start.html
>>>
>>> I propose this change because Python has become more popular than the
>>> other languages supported in Apache Spark. There are a lot more users of
>>> Spark in Python than Scala today and Python attracts a broader set of new
>>> users.
>>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava
>>> .
>>>
>>> Also, this change aligns with Python already being the first tab on our
>>> home page:
>>> https://spark.apache.org/
>>>
>>> Anyone who wants to use another language can still just click on the
>>> other tabs.
>>>
>>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>>> page as a first step:
>>> https://github.com/apache/spark/pull/40087
>>>
>>>
>>> I would appreciate it if you could share your thoughts on this proposal.
>>>
>>>
>>> Thanks a lot,
>>> Allan Folting
>>>
>>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Hyukjin Kwon <gu...@gmail.com>.
Probably it's worthwhile discussing the order for others but I would keep
it separate from this thread to focus on Python as the default since that
can be done as an incremental improvement.
On Mon, Feb 27, 2023 at 3:36 AM Mich Talebzadeh <mi...@gmail.com>
wrote:
>
> To me as I stated before this is a tangential issue. However, if it has to
> be done, we ought to work out how much effort is needed). Besides, how
> about others?
>
>
> [image: image.png]
>
> what next in the order?
>
>
>
> 1. Python
> 2. Scala
> 3. SQL
> 4. Java
> 5. R
>
> SQL is still very important and with Spark becoming more relevant to ETL
> plus lesser extend to DS, I see the above order is fair
>
> HTH
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 22 Feb 2023 at 21:00, Allan Folting <af...@gmail.com> wrote:
>
>> Hi all,
>>
>> I would like to propose that we show Python code examples first in the
>> Spark documentation where we have multiple programming language examples.
>> An example is on the Quick Start page:
>> https://spark.apache.org/docs/latest/quick-start.html
>>
>> I propose this change because Python has become more popular than the
>> other languages supported in Apache Spark. There are a lot more users of
>> Spark in Python than Scala today and Python attracts a broader set of new
>> users.
>> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
>> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
>>
>> Also, this change aligns with Python already being the first tab on our
>> home page:
>> https://spark.apache.org/
>>
>> Anyone who wants to use another language can still just click on the
>> other tabs.
>>
>> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide
>> page as a first step:
>> https://github.com/apache/spark/pull/40087
>>
>>
>> I would appreciate it if you could share your thoughts on this proposal.
>>
>>
>> Thanks a lot,
>> Allan Folting
>>
>
Re: [DISCUSS] Show Python code examples first in Spark documentation
Posted by Mich Talebzadeh <mi...@gmail.com>.
To me as I stated before this is a tangential issue. However, if it has to
be done, we ought to work out how much effort is needed). Besides, how
about others?
[image: image.png]
what next in the order?
1. Python
2. Scala
3. SQL
4. Java
5. R
SQL is still very important and with Spark becoming more relevant to ETL
plus lesser extend to DS, I see the above order is fair
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 22 Feb 2023 at 21:00, Allan Folting <af...@gmail.com> wrote:
> Hi all,
>
> I would like to propose that we show Python code examples first in the
> Spark documentation where we have multiple programming language examples.
> An example is on the Quick Start page:
> https://spark.apache.org/docs/latest/quick-start.html
>
> I propose this change because Python has become more popular than the
> other languages supported in Apache Spark. There are a lot more users of
> Spark in Python than Scala today and Python attracts a broader set of new
> users.
> For Python usage data, see https://www.tiobe.com/tiobe-index/ and
> https://insights.stackoverflow.com/trends?tags=r%2Cscala%2Cpython%2Cjava.
>
> Also, this change aligns with Python already being the first tab on our
> home page:
> https://spark.apache.org/
>
> Anyone who wants to use another language can still just click on the other
> tabs.
>
> I created a draft PR for the Spark SQL, DataFrames and Datasets Guide page
> as a first step:
> https://github.com/apache/spark/pull/40087
>
>
> I would appreciate it if you could share your thoughts on this proposal.
>
>
> Thanks a lot,
> Allan Folting
>