You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by José Müller <jo...@gmail.com> on 2025/02/10 12:02:00 UTC
[PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Hi all,
I'm new to the Spark community—please let me know if this isn’t the right
forum for feature proposals.
*About Me:*
I’ve spent over 10 years in data roles, from engineering to machine
learning and analytics. A recurring challenge I've encountered is the
disconnect between data engineering and ML teams when it comes to feature
engineering and data transformations using Spark.
*The Problem:*
There is a growing disconnect between Data Engineering and Machine Learning
teams due to differences in the tools and languages they use for data
transformations.
-
*Tooling Divergence:*
- *Data Engineering Teams* primarily use *Spark SQL* and *PySpark* for
building scalable data pipelines and transforming data across different
layers (bronze, silver, gold).
- *Machine Learning Teams* typically rely on *Pandas* for feature
engineering, model training, and recreating features on-the-fly in
production environments.
-
*Challenges Arising from This Divergence:*
1.
*Code Duplication & Maintenance Overhead:*
- ML teams often need to rebuild feature pipelines from raw data or
adapt data engineer outputs into Pandas, leading to
duplicated effort and
maintenance in two different codebases.
- This forces teams to maintain expertise in multiple frameworks,
increasing complexity and reducing cross-team supportability.
2.
*Inconsistent Feature Definitions:*
- Transformations are implemented in different languages (PySpark for
DE, Pandas for ML), which causes *feature drift*—where the same
metrics or features calculated in separate environments
diverge over time,
leading to inconsistent results.
3.
*Performance Bottlenecks in Production:*
- While PySpark can regenerate features for serving models via APIs,
it introduces inefficiencies:
- *Overhead for Small Payloads:* Spark's distributed processing
is unnecessary for small, real-time requests and is slower
compared to
Pandas.
- *Latency Issues:* Spark struggles to deliver
millisecond-level responses required in production APIs.
- *Operational Complexity:* Maintaining Spark within API
environments adds unnecessary overhead.
4.
*Data Refinement Gap:*
- Since ML teams are often formed after data engineering teams, they
lag behind in leveraging years of PySpark-based data refinements.
Reproducing the same transformations in Pandas to match the
DE pipelines is
time-consuming and error-prone.
*The Ideal Scenario*
To bridge the gap between Data Engineering and Machine Learning teams, the
ideal solution would:
1.
*Unify the Tech Stack:*
- Both Data Engineers and ML teams would use *PySpark syntax* for data
transformations and feature engineering. This shared language would
simplify code maintenance, improve collaboration, and reduce the need for
cross-training in multiple frameworks.
2.
*Flexible Execution Backend:*
- While using the same PySpark code, teams could choose the most
efficient execution engine based on their needs:
- *Data Engineers* would continue leveraging Spark's distributed
processing for large-scale data transformations.
- *ML Teams* could run the same PySpark transformations using
*Pandas* as the processing engine for faster, on-the-fly feature
generation in model training and API serving.
This unified approach would eliminate redundant codebases, ensure
consistent feature definitions, and optimize performance across both batch
and real-time workflows.
*The Proposal:*
*Introduce an API that allows PySpark syntax while processing DataFrame
using either Spark or Pandas depending on the session context.*
*Simple, but intuitive example:*
import pyspark.sql.functions as F
def silver(bronze_df):
return (
bronze_df
.withColumnRenamed("bronze_col", "silver_col")
)
def gold(silver_df):
return (
silver_df
.withColumnRenamed("silver_col", "gold_col")
.withColumn("gold_col", F.col("gold_col") + 1)
)
def features(gold_df):
return (
gold_df
.withColumnRenamed("gold_col", "feature_col")
.withColumn("feature_col", F.col("feature_col") + 1)
)
# With the Spark Session (normal way of using PySpark)
spark = SparkSession.builder.master("local[1]").getOrCreate()
bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
silver_df = silver(bronze_df)
gold_df = gold(silver_df)
features_df = features(gold_df)
features_df.show()
# Proposed "Pandas Spark Session"
spark = SparkSession.builder.as_pandas.getOrCreate()
# This would execute the same transformations using Pandas under the hood.
This would enable teams to share the same codebase while choosing the most
efficient processing engine.
We've built and experimented in different data teams with a public
library, *flyipe
<https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
that uses pandas_api() transformations, but can run using Pandas
<https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
but it still requires ML teams to manage separate pipelines for Spark
dependencies.
I’d love to hear thoughts from the community on this idea, and *if there's
a better approach to solving this issue*.
Thanks,
José Müller
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Tornike Gurgenidze <to...@freeuni.edu.ge>.
"I’d love to hear thoughts from the community on this idea, and
*if there's a better approach to solving this issue" *
In case you aren't aware, there are already tools out there (outside of
spark) that try to accomplish something similar. Probably your best options
are either ibis <https://github.com/ibis-project/ibis> or sqlframe
<https://github.com/eakmanrq/sqlframe>. ibis has it's own distinct API,
while sqlframe tries to mimic pyspark. Pandas backend is no longer
supported in either of them (ibis dropped it recently), but there are other
more performant single-node engines that you can choose from.
On Mon, Feb 10, 2025 at 6:33 PM José Müller <jo...@gmail.com>
wrote:
> Hi Wenchen,
>
> "Interesting, so this is PySpark on pandas which is the reverse of Koalas."
> yes, exactly
>
> "If performance is the only problem, maybe we can improve local-mode Spark
> performance to be on par with these single-node engines. + @Hyukjin Kwon
> <gu...@gmail.com> "
> Not sure if we could gain much performance, even in single nodes. I did
> some tests and tried to run spark as lean as possible, but the could not
> get very simple transformation like column rename or sum to run under a
> second, not an expert either, may there is a single-node setup that could
> achieve performance as fast as pandas?
>
> We have to consider the time to create a spark session in API that takes a
> considerable amount of time to create it. Users could try to minimise that
> by creating some kind of SparkSessioFactory, but it will require more from
> the users and they might not have much flexibility when hosting their
> models.
>
> Thanks for the reply.
>
> Müller
>
> On Mon, 10 Feb 2025 at 11:09, Wenchen Fan <cl...@gmail.com> wrote:
>
>> Interesting, so this is PySpark on pandas which is the reverse of Koalas.
>>
>> If performance is the only problem, maybe we can improve local-mode Spark
>> performance to be on par with these single-node engines. + @Hyukjin Kwon
>> <gu...@gmail.com>
>>
>> On Mon, Feb 10, 2025 at 8:40 PM José Müller <jo...@gmail.com>
>> wrote:
>>
>>> Hi Mitch,
>>>
>>> All you said is well understood, but I believe you are missing the
>>> point, the proposal is not to break Spark ways of processing, but to use
>>> spark as a wrapper to process pandas, same as `pandas_api()`, but the
>>> inverse.
>>>
>>> Most of the cases to serve ML models require low latency (ms) and ideal
>>> is to re-generate the features on the fly when receiving the payload via a
>>> POST request, for example.
>>>
>>> In these cases the payloads are small and do not require the whole Spark
>>> infrastructure, and it is better to process using local pandas instead of
>>> Spark, so the proposal is Spark to "map" the
>>> Pyspark Dataframes transformations and invoke Pandas transformations.
>>>
>>> I believe the point you are missing is when building the features for
>>> model training, we have large datasets and therefore Spark is recommended,
>>> but spark is not recommended to serve realtime and low latency requests.
>>>
>>> I know Koalas and pandas api, as described in the email, at the end.
>>> However using koalas or pandas does not solve the problem of data teams
>>> having to support 2 code syntaxes.
>>>
>>> Müller
>>>
>>> On Mon, 10 Feb 2025 at 09:27, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> Regardless there are technical limitations here. For example pandas
>>>> operates in-memory on a single machine like the driver, making it
>>>> unsuitable for large-scale datasets that exceed the memory or processing
>>>> capacity of a single node. Unlike pySpark, pandas cannot distribute
>>>> workloads across a cluster, Have you looked at Koalas which I believe
>>>> is currently integrated as pyspark.pandas?
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, 10 Feb 2025 at 12:04, José Müller <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm new to the Spark community—please let me know if this isn’t the
>>>>> right forum for feature proposals.
>>>>>
>>>>> *About Me:*
>>>>> I’ve spent over 10 years in data roles, from engineering to machine
>>>>> learning and analytics. A recurring challenge I've encountered is the
>>>>> disconnect between data engineering and ML teams when it comes to feature
>>>>> engineering and data transformations using Spark.
>>>>>
>>>>> *The Problem:*
>>>>>
>>>>> There is a growing disconnect between Data Engineering and Machine
>>>>> Learning teams due to differences in the tools and languages they use for
>>>>> data transformations.
>>>>>
>>>>> -
>>>>>
>>>>> *Tooling Divergence:*
>>>>> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark*
>>>>> for building scalable data pipelines and transforming data across different
>>>>> layers (bronze, silver, gold).
>>>>> - *Machine Learning Teams* typically rely on *Pandas* for
>>>>> feature engineering, model training, and recreating features on-the-fly in
>>>>> production environments.
>>>>> -
>>>>>
>>>>> *Challenges Arising from This Divergence:*
>>>>> 1.
>>>>>
>>>>> *Code Duplication & Maintenance Overhead:*
>>>>> - ML teams often need to rebuild feature pipelines from raw
>>>>> data or adapt data engineer outputs into Pandas, leading to duplicated
>>>>> effort and maintenance in two different codebases.
>>>>> - This forces teams to maintain expertise in multiple
>>>>> frameworks, increasing complexity and reducing cross-team supportability.
>>>>> 2.
>>>>>
>>>>> *Inconsistent Feature Definitions:*
>>>>> - Transformations are implemented in different languages
>>>>> (PySpark for DE, Pandas for ML), which causes *feature drift*—where
>>>>> the same metrics or features calculated in separate environments diverge
>>>>> over time, leading to inconsistent results.
>>>>> 3.
>>>>>
>>>>> *Performance Bottlenecks in Production:*
>>>>> - While PySpark can regenerate features for serving models via
>>>>> APIs, it introduces inefficiencies:
>>>>> - *Overhead for Small Payloads:* Spark's distributed
>>>>> processing is unnecessary for small, real-time requests and is slower
>>>>> compared to Pandas.
>>>>> - *Latency Issues:* Spark struggles to deliver
>>>>> millisecond-level responses required in production APIs.
>>>>> - *Operational Complexity:* Maintaining Spark within API
>>>>> environments adds unnecessary overhead.
>>>>> 4.
>>>>>
>>>>> *Data Refinement Gap:*
>>>>> - Since ML teams are often formed after data engineering teams,
>>>>> they lag behind in leveraging years of PySpark-based data refinements.
>>>>> Reproducing the same transformations in Pandas to match the DE pipelines is
>>>>> time-consuming and error-prone.
>>>>>
>>>>>
>>>>> *The Ideal Scenario*
>>>>>
>>>>> To bridge the gap between Data Engineering and Machine Learning teams,
>>>>> the ideal solution would:
>>>>>
>>>>> 1.
>>>>>
>>>>> *Unify the Tech Stack:*
>>>>> - Both Data Engineers and ML teams would use *PySpark syntax* for
>>>>> data transformations and feature engineering. This shared language would
>>>>> simplify code maintenance, improve collaboration, and reduce the need for
>>>>> cross-training in multiple frameworks.
>>>>> 2.
>>>>>
>>>>> *Flexible Execution Backend:*
>>>>> - While using the same PySpark code, teams could choose the most
>>>>> efficient execution engine based on their needs:
>>>>> - *Data Engineers* would continue leveraging Spark's
>>>>> distributed processing for large-scale data transformations.
>>>>> - *ML Teams* could run the same PySpark transformations
>>>>> using *Pandas* as the processing engine for faster,
>>>>> on-the-fly feature generation in model training and API serving.
>>>>>
>>>>> This unified approach would eliminate redundant codebases, ensure
>>>>> consistent feature definitions, and optimize performance across both batch
>>>>> and real-time workflows.
>>>>>
>>>>>
>>>>> *The Proposal:*
>>>>>
>>>>> *Introduce an API that allows PySpark syntax while processing
>>>>> DataFrame using either Spark or Pandas depending on the session context.*
>>>>>
>>>>>
>>>>> *Simple, but intuitive example:*
>>>>>
>>>>> import pyspark.sql.functions as F
>>>>>
>>>>> def silver(bronze_df):
>>>>> return (
>>>>> bronze_df
>>>>> .withColumnRenamed("bronze_col", "silver_col")
>>>>> )
>>>>>
>>>>> def gold(silver_df):
>>>>> return (
>>>>> silver_df
>>>>> .withColumnRenamed("silver_col", "gold_col")
>>>>> .withColumn("gold_col", F.col("gold_col") + 1)
>>>>> )
>>>>>
>>>>> def features(gold_df):
>>>>> return (
>>>>> gold_df
>>>>> .withColumnRenamed("gold_col", "feature_col")
>>>>> .withColumn("feature_col", F.col("feature_col") + 1)
>>>>> )
>>>>>
>>>>> # With the Spark Session (normal way of using PySpark)
>>>>> spark = SparkSession.builder.master("local[1]").getOrCreate()
>>>>> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
>>>>> silver_df = silver(bronze_df)
>>>>> gold_df = gold(silver_df)
>>>>> features_df = features(gold_df)
>>>>> features_df.show()
>>>>>
>>>>> # Proposed "Pandas Spark Session"
>>>>> spark = SparkSession.builder.as_pandas.getOrCreate()
>>>>> # This would execute the same transformations using Pandas under the hood.
>>>>>
>>>>> This would enable teams to share the same codebase while choosing the
>>>>> most efficient processing engine.
>>>>>
>>>>> We've built and experimented in different data teams with a public
>>>>> library, *flyipe
>>>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
>>>>> that uses pandas_api() transformations, but can run using Pandas
>>>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
>>>>> but it still requires ML teams to manage separate pipelines for Spark
>>>>> dependencies.
>>>>>
>>>>> I’d love to hear thoughts from the community on this idea, and *if
>>>>> there's a better approach to solving this issue*.
>>>>>
>>>>> Thanks,
>>>>> José Müller
>>>>>
>>>>
>>>
>>> --
>>> José Müller
>>>
>>
>
> --
> José Müller
>
--
Tornike Gurgenidze
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by José Müller <jo...@gmail.com>.
Hi Wenchen,
"Interesting, so this is PySpark on pandas which is the reverse of Koalas."
yes, exactly
"If performance is the only problem, maybe we can improve local-mode Spark
performance to be on par with these single-node engines. + @Hyukjin Kwon
<gu...@gmail.com> "
Not sure if we could gain much performance, even in single nodes. I did
some tests and tried to run spark as lean as possible, but the could not
get very simple transformation like column rename or sum to run under a
second, not an expert either, may there is a single-node setup that could
achieve performance as fast as pandas?
We have to consider the time to create a spark session in API that takes a
considerable amount of time to create it. Users could try to minimise that
by creating some kind of SparkSessioFactory, but it will require more from
the users and they might not have much flexibility when hosting their
models.
Thanks for the reply.
Müller
On Mon, 10 Feb 2025 at 11:09, Wenchen Fan <cl...@gmail.com> wrote:
> Interesting, so this is PySpark on pandas which is the reverse of Koalas.
>
> If performance is the only problem, maybe we can improve local-mode Spark
> performance to be on par with these single-node engines. + @Hyukjin Kwon
> <gu...@gmail.com>
>
> On Mon, Feb 10, 2025 at 8:40 PM José Müller <jo...@gmail.com>
> wrote:
>
>> Hi Mitch,
>>
>> All you said is well understood, but I believe you are missing the point,
>> the proposal is not to break Spark ways of processing, but to use spark as
>> a wrapper to process pandas, same as `pandas_api()`, but the inverse.
>>
>> Most of the cases to serve ML models require low latency (ms) and ideal
>> is to re-generate the features on the fly when receiving the payload via a
>> POST request, for example.
>>
>> In these cases the payloads are small and do not require the whole Spark
>> infrastructure, and it is better to process using local pandas instead of
>> Spark, so the proposal is Spark to "map" the
>> Pyspark Dataframes transformations and invoke Pandas transformations.
>>
>> I believe the point you are missing is when building the features for
>> model training, we have large datasets and therefore Spark is recommended,
>> but spark is not recommended to serve realtime and low latency requests.
>>
>> I know Koalas and pandas api, as described in the email, at the end.
>> However using koalas or pandas does not solve the problem of data teams
>> having to support 2 code syntaxes.
>>
>> Müller
>>
>> On Mon, 10 Feb 2025 at 09:27, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Regardless there are technical limitations here. For example pandas
>>> operates in-memory on a single machine like the driver, making it
>>> unsuitable for large-scale datasets that exceed the memory or processing
>>> capacity of a single node. Unlike pySpark, pandas cannot distribute
>>> workloads across a cluster, Have you looked at Koalas which I believe
>>> is currently integrated as pyspark.pandas?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, 10 Feb 2025 at 12:04, José Müller <jo...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm new to the Spark community—please let me know if this isn’t the
>>>> right forum for feature proposals.
>>>>
>>>> *About Me:*
>>>> I’ve spent over 10 years in data roles, from engineering to machine
>>>> learning and analytics. A recurring challenge I've encountered is the
>>>> disconnect between data engineering and ML teams when it comes to feature
>>>> engineering and data transformations using Spark.
>>>>
>>>> *The Problem:*
>>>>
>>>> There is a growing disconnect between Data Engineering and Machine
>>>> Learning teams due to differences in the tools and languages they use for
>>>> data transformations.
>>>>
>>>> -
>>>>
>>>> *Tooling Divergence:*
>>>> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark*
>>>> for building scalable data pipelines and transforming data across different
>>>> layers (bronze, silver, gold).
>>>> - *Machine Learning Teams* typically rely on *Pandas* for
>>>> feature engineering, model training, and recreating features on-the-fly in
>>>> production environments.
>>>> -
>>>>
>>>> *Challenges Arising from This Divergence:*
>>>> 1.
>>>>
>>>> *Code Duplication & Maintenance Overhead:*
>>>> - ML teams often need to rebuild feature pipelines from raw data
>>>> or adapt data engineer outputs into Pandas, leading to duplicated effort
>>>> and maintenance in two different codebases.
>>>> - This forces teams to maintain expertise in multiple
>>>> frameworks, increasing complexity and reducing cross-team supportability.
>>>> 2.
>>>>
>>>> *Inconsistent Feature Definitions:*
>>>> - Transformations are implemented in different languages
>>>> (PySpark for DE, Pandas for ML), which causes *feature drift*—where
>>>> the same metrics or features calculated in separate environments diverge
>>>> over time, leading to inconsistent results.
>>>> 3.
>>>>
>>>> *Performance Bottlenecks in Production:*
>>>> - While PySpark can regenerate features for serving models via
>>>> APIs, it introduces inefficiencies:
>>>> - *Overhead for Small Payloads:* Spark's distributed
>>>> processing is unnecessary for small, real-time requests and is slower
>>>> compared to Pandas.
>>>> - *Latency Issues:* Spark struggles to deliver
>>>> millisecond-level responses required in production APIs.
>>>> - *Operational Complexity:* Maintaining Spark within API
>>>> environments adds unnecessary overhead.
>>>> 4.
>>>>
>>>> *Data Refinement Gap:*
>>>> - Since ML teams are often formed after data engineering teams,
>>>> they lag behind in leveraging years of PySpark-based data refinements.
>>>> Reproducing the same transformations in Pandas to match the DE pipelines is
>>>> time-consuming and error-prone.
>>>>
>>>>
>>>> *The Ideal Scenario*
>>>>
>>>> To bridge the gap between Data Engineering and Machine Learning teams,
>>>> the ideal solution would:
>>>>
>>>> 1.
>>>>
>>>> *Unify the Tech Stack:*
>>>> - Both Data Engineers and ML teams would use *PySpark syntax* for
>>>> data transformations and feature engineering. This shared language would
>>>> simplify code maintenance, improve collaboration, and reduce the need for
>>>> cross-training in multiple frameworks.
>>>> 2.
>>>>
>>>> *Flexible Execution Backend:*
>>>> - While using the same PySpark code, teams could choose the most
>>>> efficient execution engine based on their needs:
>>>> - *Data Engineers* would continue leveraging Spark's
>>>> distributed processing for large-scale data transformations.
>>>> - *ML Teams* could run the same PySpark transformations using
>>>> *Pandas* as the processing engine for faster, on-the-fly
>>>> feature generation in model training and API serving.
>>>>
>>>> This unified approach would eliminate redundant codebases, ensure
>>>> consistent feature definitions, and optimize performance across both batch
>>>> and real-time workflows.
>>>>
>>>>
>>>> *The Proposal:*
>>>>
>>>> *Introduce an API that allows PySpark syntax while processing DataFrame
>>>> using either Spark or Pandas depending on the session context.*
>>>>
>>>>
>>>> *Simple, but intuitive example:*
>>>>
>>>> import pyspark.sql.functions as F
>>>>
>>>> def silver(bronze_df):
>>>> return (
>>>> bronze_df
>>>> .withColumnRenamed("bronze_col", "silver_col")
>>>> )
>>>>
>>>> def gold(silver_df):
>>>> return (
>>>> silver_df
>>>> .withColumnRenamed("silver_col", "gold_col")
>>>> .withColumn("gold_col", F.col("gold_col") + 1)
>>>> )
>>>>
>>>> def features(gold_df):
>>>> return (
>>>> gold_df
>>>> .withColumnRenamed("gold_col", "feature_col")
>>>> .withColumn("feature_col", F.col("feature_col") + 1)
>>>> )
>>>>
>>>> # With the Spark Session (normal way of using PySpark)
>>>> spark = SparkSession.builder.master("local[1]").getOrCreate()
>>>> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
>>>> silver_df = silver(bronze_df)
>>>> gold_df = gold(silver_df)
>>>> features_df = features(gold_df)
>>>> features_df.show()
>>>>
>>>> # Proposed "Pandas Spark Session"
>>>> spark = SparkSession.builder.as_pandas.getOrCreate()
>>>> # This would execute the same transformations using Pandas under the hood.
>>>>
>>>> This would enable teams to share the same codebase while choosing the
>>>> most efficient processing engine.
>>>>
>>>> We've built and experimented in different data teams with a public
>>>> library, *flyipe
>>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
>>>> that uses pandas_api() transformations, but can run using Pandas
>>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
>>>> but it still requires ML teams to manage separate pipelines for Spark
>>>> dependencies.
>>>>
>>>> I’d love to hear thoughts from the community on this idea, and *if
>>>> there's a better approach to solving this issue*.
>>>>
>>>> Thanks,
>>>> José Müller
>>>>
>>>
>>
>> --
>> José Müller
>>
>
--
José Müller
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Wenchen Fan <cl...@gmail.com>.
Interesting, so this is PySpark on pandas which is the reverse of Koalas.
If performance is the only problem, maybe we can improve local-mode Spark
performance to be on par with these single-node engines. + @Hyukjin Kwon
<gu...@gmail.com>
On Mon, Feb 10, 2025 at 8:40 PM José Müller <jo...@gmail.com>
wrote:
> Hi Mitch,
>
> All you said is well understood, but I believe you are missing the point,
> the proposal is not to break Spark ways of processing, but to use spark as
> a wrapper to process pandas, same as `pandas_api()`, but the inverse.
>
> Most of the cases to serve ML models require low latency (ms) and ideal is
> to re-generate the features on the fly when receiving the payload via a
> POST request, for example.
>
> In these cases the payloads are small and do not require the whole Spark
> infrastructure, and it is better to process using local pandas instead of
> Spark, so the proposal is Spark to "map" the
> Pyspark Dataframes transformations and invoke Pandas transformations.
>
> I believe the point you are missing is when building the features for
> model training, we have large datasets and therefore Spark is recommended,
> but spark is not recommended to serve realtime and low latency requests.
>
> I know Koalas and pandas api, as described in the email, at the end.
> However using koalas or pandas does not solve the problem of data teams
> having to support 2 code syntaxes.
>
> Müller
>
> On Mon, 10 Feb 2025 at 09:27, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Regardless there are technical limitations here. For example pandas
>> operates in-memory on a single machine like the driver, making it
>> unsuitable for large-scale datasets that exceed the memory or processing
>> capacity of a single node. Unlike pySpark, pandas cannot distribute
>> workloads across a cluster, Have you looked at Koalas which I believe is
>> currently integrated as pyspark.pandas?
>>
>> HTH
>>
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Mon, 10 Feb 2025 at 12:04, José Müller <jo...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I'm new to the Spark community—please let me know if this isn’t the
>>> right forum for feature proposals.
>>>
>>> *About Me:*
>>> I’ve spent over 10 years in data roles, from engineering to machine
>>> learning and analytics. A recurring challenge I've encountered is the
>>> disconnect between data engineering and ML teams when it comes to feature
>>> engineering and data transformations using Spark.
>>>
>>> *The Problem:*
>>>
>>> There is a growing disconnect between Data Engineering and Machine
>>> Learning teams due to differences in the tools and languages they use for
>>> data transformations.
>>>
>>> -
>>>
>>> *Tooling Divergence:*
>>> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark*
>>> for building scalable data pipelines and transforming data across different
>>> layers (bronze, silver, gold).
>>> - *Machine Learning Teams* typically rely on *Pandas* for feature
>>> engineering, model training, and recreating features on-the-fly in
>>> production environments.
>>> -
>>>
>>> *Challenges Arising from This Divergence:*
>>> 1.
>>>
>>> *Code Duplication & Maintenance Overhead:*
>>> - ML teams often need to rebuild feature pipelines from raw data
>>> or adapt data engineer outputs into Pandas, leading to duplicated effort
>>> and maintenance in two different codebases.
>>> - This forces teams to maintain expertise in multiple
>>> frameworks, increasing complexity and reducing cross-team supportability.
>>> 2.
>>>
>>> *Inconsistent Feature Definitions:*
>>> - Transformations are implemented in different languages (PySpark
>>> for DE, Pandas for ML), which causes *feature drift*—where the
>>> same metrics or features calculated in separate environments diverge over
>>> time, leading to inconsistent results.
>>> 3.
>>>
>>> *Performance Bottlenecks in Production:*
>>> - While PySpark can regenerate features for serving models via
>>> APIs, it introduces inefficiencies:
>>> - *Overhead for Small Payloads:* Spark's distributed
>>> processing is unnecessary for small, real-time requests and is slower
>>> compared to Pandas.
>>> - *Latency Issues:* Spark struggles to deliver
>>> millisecond-level responses required in production APIs.
>>> - *Operational Complexity:* Maintaining Spark within API
>>> environments adds unnecessary overhead.
>>> 4.
>>>
>>> *Data Refinement Gap:*
>>> - Since ML teams are often formed after data engineering teams,
>>> they lag behind in leveraging years of PySpark-based data refinements.
>>> Reproducing the same transformations in Pandas to match the DE pipelines is
>>> time-consuming and error-prone.
>>>
>>>
>>> *The Ideal Scenario*
>>>
>>> To bridge the gap between Data Engineering and Machine Learning teams,
>>> the ideal solution would:
>>>
>>> 1.
>>>
>>> *Unify the Tech Stack:*
>>> - Both Data Engineers and ML teams would use *PySpark syntax* for
>>> data transformations and feature engineering. This shared language would
>>> simplify code maintenance, improve collaboration, and reduce the need for
>>> cross-training in multiple frameworks.
>>> 2.
>>>
>>> *Flexible Execution Backend:*
>>> - While using the same PySpark code, teams could choose the most
>>> efficient execution engine based on their needs:
>>> - *Data Engineers* would continue leveraging Spark's
>>> distributed processing for large-scale data transformations.
>>> - *ML Teams* could run the same PySpark transformations using
>>> *Pandas* as the processing engine for faster, on-the-fly
>>> feature generation in model training and API serving.
>>>
>>> This unified approach would eliminate redundant codebases, ensure
>>> consistent feature definitions, and optimize performance across both batch
>>> and real-time workflows.
>>>
>>>
>>> *The Proposal:*
>>>
>>> *Introduce an API that allows PySpark syntax while processing DataFrame
>>> using either Spark or Pandas depending on the session context.*
>>>
>>>
>>> *Simple, but intuitive example:*
>>>
>>> import pyspark.sql.functions as F
>>>
>>> def silver(bronze_df):
>>> return (
>>> bronze_df
>>> .withColumnRenamed("bronze_col", "silver_col")
>>> )
>>>
>>> def gold(silver_df):
>>> return (
>>> silver_df
>>> .withColumnRenamed("silver_col", "gold_col")
>>> .withColumn("gold_col", F.col("gold_col") + 1)
>>> )
>>>
>>> def features(gold_df):
>>> return (
>>> gold_df
>>> .withColumnRenamed("gold_col", "feature_col")
>>> .withColumn("feature_col", F.col("feature_col") + 1)
>>> )
>>>
>>> # With the Spark Session (normal way of using PySpark)
>>> spark = SparkSession.builder.master("local[1]").getOrCreate()
>>> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
>>> silver_df = silver(bronze_df)
>>> gold_df = gold(silver_df)
>>> features_df = features(gold_df)
>>> features_df.show()
>>>
>>> # Proposed "Pandas Spark Session"
>>> spark = SparkSession.builder.as_pandas.getOrCreate()
>>> # This would execute the same transformations using Pandas under the hood.
>>>
>>> This would enable teams to share the same codebase while choosing the
>>> most efficient processing engine.
>>>
>>> We've built and experimented in different data teams with a public
>>> library, *flyipe
>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
>>> that uses pandas_api() transformations, but can run using Pandas
>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
>>> but it still requires ML teams to manage separate pipelines for Spark
>>> dependencies.
>>>
>>> I’d love to hear thoughts from the community on this idea, and *if
>>> there's a better approach to solving this issue*.
>>>
>>> Thanks,
>>> José Müller
>>>
>>
>
> --
> José Müller
>
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Mich Talebzadeh <mi...@gmail.com>.
I would suggest using some feature like Lambda Architecture which is
flexible enough. This is a dated diagram that illustrates better
The architecture will consist of two main components plus Mapping
1) Batch Feature Engineering with Spark: Purpose: Process large datasets to
generate features for model training
- Use Spark for large-scale feature engineering during model training.
- Store the feature engineering logic in a reusable format (e.g., a
transformation pipeline).
2) Real-Time Feature Generation with Pandas: Purpose: Generate features
on-the-fly for low-latency model serving
- Use Pandas for low-latency, on-the-fly feature generation during model
serving.
- Translate Spark transformations into Pandas transformations for
consistency.
3) Mapping Spark Transformations to Pandas: Purpose: Ensure consistency
between batch and real-time feature engineering
- Extract the transformation logic from the Spark pipeline.
- Implement equivalent transformations in Pandas.
- Use a shared library or configuration file to maintain consistency
between Spark and Pandas transformations.
[image: LambdaArchitecture_MD_BigQuery.png]
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 10 Feb 2025 at 12:39, José Müller <jo...@gmail.com> wrote:
> Hi Mitch,
>
> All you said is well understood, but I believe you are missing the point,
> the proposal is not to break Spark ways of processing, but to use spark as
> a wrapper to process pandas, same as `pandas_api()`, but the inverse.
>
> Most of the cases to serve ML models require low latency (ms) and ideal is
> to re-generate the features on the fly when receiving the payload via a
> POST request, for example.
>
> In these cases the payloads are small and do not require the whole Spark
> infrastructure, and it is better to process using local pandas instead of
> Spark, so the proposal is Spark to "map" the
> Pyspark Dataframes transformations and invoke Pandas transformations.
>
> I believe the point you are missing is when building the features for
> model training, we have large datasets and therefore Spark is recommended,
> but spark is not recommended to serve realtime and low latency requests.
>
> I know Koalas and pandas api, as described in the email, at the end.
> However using koalas or pandas does not solve the problem of data teams
> having to support 2 code syntaxes.
>
> Müller
>
> On Mon, 10 Feb 2025 at 09:27, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Regardless there are technical limitations here. For example pandas
>> operates in-memory on a single machine like the driver, making it
>> unsuitable for large-scale datasets that exceed the memory or processing
>> capacity of a single node. Unlike pySpark, pandas cannot distribute
>> workloads across a cluster, Have you looked at Koalas which I believe is
>> currently integrated as pyspark.pandas?
>>
>> HTH
>>
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Mon, 10 Feb 2025 at 12:04, José Müller <jo...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I'm new to the Spark community—please let me know if this isn’t the
>>> right forum for feature proposals.
>>>
>>> *About Me:*
>>> I’ve spent over 10 years in data roles, from engineering to machine
>>> learning and analytics. A recurring challenge I've encountered is the
>>> disconnect between data engineering and ML teams when it comes to feature
>>> engineering and data transformations using Spark.
>>>
>>> *The Problem:*
>>>
>>> There is a growing disconnect between Data Engineering and Machine
>>> Learning teams due to differences in the tools and languages they use for
>>> data transformations.
>>>
>>> -
>>>
>>> *Tooling Divergence:*
>>> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark*
>>> for building scalable data pipelines and transforming data across different
>>> layers (bronze, silver, gold).
>>> - *Machine Learning Teams* typically rely on *Pandas* for feature
>>> engineering, model training, and recreating features on-the-fly in
>>> production environments.
>>> -
>>>
>>> *Challenges Arising from This Divergence:*
>>> 1.
>>>
>>> *Code Duplication & Maintenance Overhead:*
>>> - ML teams often need to rebuild feature pipelines from raw data
>>> or adapt data engineer outputs into Pandas, leading to duplicated effort
>>> and maintenance in two different codebases.
>>> - This forces teams to maintain expertise in multiple
>>> frameworks, increasing complexity and reducing cross-team supportability.
>>> 2.
>>>
>>> *Inconsistent Feature Definitions:*
>>> - Transformations are implemented in different languages (PySpark
>>> for DE, Pandas for ML), which causes *feature drift*—where the
>>> same metrics or features calculated in separate environments diverge over
>>> time, leading to inconsistent results.
>>> 3.
>>>
>>> *Performance Bottlenecks in Production:*
>>> - While PySpark can regenerate features for serving models via
>>> APIs, it introduces inefficiencies:
>>> - *Overhead for Small Payloads:* Spark's distributed
>>> processing is unnecessary for small, real-time requests and is slower
>>> compared to Pandas.
>>> - *Latency Issues:* Spark struggles to deliver
>>> millisecond-level responses required in production APIs.
>>> - *Operational Complexity:* Maintaining Spark within API
>>> environments adds unnecessary overhead.
>>> 4.
>>>
>>> *Data Refinement Gap:*
>>> - Since ML teams are often formed after data engineering teams,
>>> they lag behind in leveraging years of PySpark-based data refinements.
>>> Reproducing the same transformations in Pandas to match the DE pipelines is
>>> time-consuming and error-prone.
>>>
>>>
>>> *The Ideal Scenario*
>>>
>>> To bridge the gap between Data Engineering and Machine Learning teams,
>>> the ideal solution would:
>>>
>>> 1.
>>>
>>> *Unify the Tech Stack:*
>>> - Both Data Engineers and ML teams would use *PySpark syntax* for
>>> data transformations and feature engineering. This shared language would
>>> simplify code maintenance, improve collaboration, and reduce the need for
>>> cross-training in multiple frameworks.
>>> 2.
>>>
>>> *Flexible Execution Backend:*
>>> - While using the same PySpark code, teams could choose the most
>>> efficient execution engine based on their needs:
>>> - *Data Engineers* would continue leveraging Spark's
>>> distributed processing for large-scale data transformations.
>>> - *ML Teams* could run the same PySpark transformations using
>>> *Pandas* as the processing engine for faster, on-the-fly
>>> feature generation in model training and API serving.
>>>
>>> This unified approach would eliminate redundant codebases, ensure
>>> consistent feature definitions, and optimize performance across both batch
>>> and real-time workflows.
>>>
>>>
>>> *The Proposal:*
>>>
>>> *Introduce an API that allows PySpark syntax while processing DataFrame
>>> using either Spark or Pandas depending on the session context.*
>>>
>>>
>>> *Simple, but intuitive example:*
>>>
>>> import pyspark.sql.functions as F
>>>
>>> def silver(bronze_df):
>>> return (
>>> bronze_df
>>> .withColumnRenamed("bronze_col", "silver_col")
>>> )
>>>
>>> def gold(silver_df):
>>> return (
>>> silver_df
>>> .withColumnRenamed("silver_col", "gold_col")
>>> .withColumn("gold_col", F.col("gold_col") + 1)
>>> )
>>>
>>> def features(gold_df):
>>> return (
>>> gold_df
>>> .withColumnRenamed("gold_col", "feature_col")
>>> .withColumn("feature_col", F.col("feature_col") + 1)
>>> )
>>>
>>> # With the Spark Session (normal way of using PySpark)
>>> spark = SparkSession.builder.master("local[1]").getOrCreate()
>>> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
>>> silver_df = silver(bronze_df)
>>> gold_df = gold(silver_df)
>>> features_df = features(gold_df)
>>> features_df.show()
>>>
>>> # Proposed "Pandas Spark Session"
>>> spark = SparkSession.builder.as_pandas.getOrCreate()
>>> # This would execute the same transformations using Pandas under the hood.
>>>
>>> This would enable teams to share the same codebase while choosing the
>>> most efficient processing engine.
>>>
>>> We've built and experimented in different data teams with a public
>>> library, *flyipe
>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
>>> that uses pandas_api() transformations, but can run using Pandas
>>> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
>>> but it still requires ML teams to manage separate pipelines for Spark
>>> dependencies.
>>>
>>> I’d love to hear thoughts from the community on this idea, and *if
>>> there's a better approach to solving this issue*.
>>>
>>> Thanks,
>>> José Müller
>>>
>>
>
> --
> José Müller
>
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by José Müller <jo...@gmail.com>.
Hi Mitch,
All you said is well understood, but I believe you are missing the point,
the proposal is not to break Spark ways of processing, but to use spark as
a wrapper to process pandas, same as `pandas_api()`, but the inverse.
Most of the cases to serve ML models require low latency (ms) and ideal is
to re-generate the features on the fly when receiving the payload via a
POST request, for example.
In these cases the payloads are small and do not require the whole Spark
infrastructure, and it is better to process using local pandas instead of
Spark, so the proposal is Spark to "map" the
Pyspark Dataframes transformations and invoke Pandas transformations.
I believe the point you are missing is when building the features for model
training, we have large datasets and therefore Spark is recommended, but
spark is not recommended to serve realtime and low latency requests.
I know Koalas and pandas api, as described in the email, at the end.
However using koalas or pandas does not solve the problem of data teams
having to support 2 code syntaxes.
Müller
On Mon, 10 Feb 2025 at 09:27, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Regardless there are technical limitations here. For example pandas
> operates in-memory on a single machine like the driver, making it
> unsuitable for large-scale datasets that exceed the memory or processing
> capacity of a single node. Unlike pySpark, pandas cannot distribute
> workloads across a cluster, Have you looked at Koalas which I believe is
> currently integrated as pyspark.pandas?
>
> HTH
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Mon, 10 Feb 2025 at 12:04, José Müller <jo...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I'm new to the Spark community—please let me know if this isn’t the right
>> forum for feature proposals.
>>
>> *About Me:*
>> I’ve spent over 10 years in data roles, from engineering to machine
>> learning and analytics. A recurring challenge I've encountered is the
>> disconnect between data engineering and ML teams when it comes to feature
>> engineering and data transformations using Spark.
>>
>> *The Problem:*
>>
>> There is a growing disconnect between Data Engineering and Machine
>> Learning teams due to differences in the tools and languages they use for
>> data transformations.
>>
>> -
>>
>> *Tooling Divergence:*
>> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark*
>> for building scalable data pipelines and transforming data across different
>> layers (bronze, silver, gold).
>> - *Machine Learning Teams* typically rely on *Pandas* for feature
>> engineering, model training, and recreating features on-the-fly in
>> production environments.
>> -
>>
>> *Challenges Arising from This Divergence:*
>> 1.
>>
>> *Code Duplication & Maintenance Overhead:*
>> - ML teams often need to rebuild feature pipelines from raw data
>> or adapt data engineer outputs into Pandas, leading to duplicated effort
>> and maintenance in two different codebases.
>> - This forces teams to maintain expertise in multiple
>> frameworks, increasing complexity and reducing cross-team supportability.
>> 2.
>>
>> *Inconsistent Feature Definitions:*
>> - Transformations are implemented in different languages (PySpark
>> for DE, Pandas for ML), which causes *feature drift*—where the
>> same metrics or features calculated in separate environments diverge over
>> time, leading to inconsistent results.
>> 3.
>>
>> *Performance Bottlenecks in Production:*
>> - While PySpark can regenerate features for serving models via
>> APIs, it introduces inefficiencies:
>> - *Overhead for Small Payloads:* Spark's distributed
>> processing is unnecessary for small, real-time requests and is slower
>> compared to Pandas.
>> - *Latency Issues:* Spark struggles to deliver
>> millisecond-level responses required in production APIs.
>> - *Operational Complexity:* Maintaining Spark within API
>> environments adds unnecessary overhead.
>> 4.
>>
>> *Data Refinement Gap:*
>> - Since ML teams are often formed after data engineering teams,
>> they lag behind in leveraging years of PySpark-based data refinements.
>> Reproducing the same transformations in Pandas to match the DE pipelines is
>> time-consuming and error-prone.
>>
>>
>> *The Ideal Scenario*
>>
>> To bridge the gap between Data Engineering and Machine Learning teams,
>> the ideal solution would:
>>
>> 1.
>>
>> *Unify the Tech Stack:*
>> - Both Data Engineers and ML teams would use *PySpark syntax* for
>> data transformations and feature engineering. This shared language would
>> simplify code maintenance, improve collaboration, and reduce the need for
>> cross-training in multiple frameworks.
>> 2.
>>
>> *Flexible Execution Backend:*
>> - While using the same PySpark code, teams could choose the most
>> efficient execution engine based on their needs:
>> - *Data Engineers* would continue leveraging Spark's
>> distributed processing for large-scale data transformations.
>> - *ML Teams* could run the same PySpark transformations using
>> *Pandas* as the processing engine for faster, on-the-fly
>> feature generation in model training and API serving.
>>
>> This unified approach would eliminate redundant codebases, ensure
>> consistent feature definitions, and optimize performance across both batch
>> and real-time workflows.
>>
>>
>> *The Proposal:*
>>
>> *Introduce an API that allows PySpark syntax while processing DataFrame
>> using either Spark or Pandas depending on the session context.*
>>
>>
>> *Simple, but intuitive example:*
>>
>> import pyspark.sql.functions as F
>>
>> def silver(bronze_df):
>> return (
>> bronze_df
>> .withColumnRenamed("bronze_col", "silver_col")
>> )
>>
>> def gold(silver_df):
>> return (
>> silver_df
>> .withColumnRenamed("silver_col", "gold_col")
>> .withColumn("gold_col", F.col("gold_col") + 1)
>> )
>>
>> def features(gold_df):
>> return (
>> gold_df
>> .withColumnRenamed("gold_col", "feature_col")
>> .withColumn("feature_col", F.col("feature_col") + 1)
>> )
>>
>> # With the Spark Session (normal way of using PySpark)
>> spark = SparkSession.builder.master("local[1]").getOrCreate()
>> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
>> silver_df = silver(bronze_df)
>> gold_df = gold(silver_df)
>> features_df = features(gold_df)
>> features_df.show()
>>
>> # Proposed "Pandas Spark Session"
>> spark = SparkSession.builder.as_pandas.getOrCreate()
>> # This would execute the same transformations using Pandas under the hood.
>>
>> This would enable teams to share the same codebase while choosing the
>> most efficient processing engine.
>>
>> We've built and experimented in different data teams with a public
>> library, *flyipe
>> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
>> that uses pandas_api() transformations, but can run using Pandas
>> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
>> but it still requires ML teams to manage separate pipelines for Spark
>> dependencies.
>>
>> I’d love to hear thoughts from the community on this idea, and *if
>> there's a better approach to solving this issue*.
>>
>> Thanks,
>> José Müller
>>
>
--
José Müller
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Mich Talebzadeh <mi...@gmail.com>.
Regardless there are technical limitations here. For example pandas
operates in-memory on a single machine like the driver, making it
unsuitable for large-scale datasets that exceed the memory or processing
capacity of a single node. Unlike pySpark, pandas cannot distribute
workloads across a cluster, Have you looked at Koalas which I believe is
currently integrated as pyspark.pandas?
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 10 Feb 2025 at 12:04, José Müller <jo...@gmail.com> wrote:
> Hi all,
>
> I'm new to the Spark community—please let me know if this isn’t the right
> forum for feature proposals.
>
> *About Me:*
> I’ve spent over 10 years in data roles, from engineering to machine
> learning and analytics. A recurring challenge I've encountered is the
> disconnect between data engineering and ML teams when it comes to feature
> engineering and data transformations using Spark.
>
> *The Problem:*
>
> There is a growing disconnect between Data Engineering and Machine
> Learning teams due to differences in the tools and languages they use for
> data transformations.
>
> -
>
> *Tooling Divergence:*
> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark* for
> building scalable data pipelines and transforming data across different
> layers (bronze, silver, gold).
> - *Machine Learning Teams* typically rely on *Pandas* for feature
> engineering, model training, and recreating features on-the-fly in
> production environments.
> -
>
> *Challenges Arising from This Divergence:*
> 1.
>
> *Code Duplication & Maintenance Overhead:*
> - ML teams often need to rebuild feature pipelines from raw data or
> adapt data engineer outputs into Pandas, leading to duplicated effort and
> maintenance in two different codebases.
> - This forces teams to maintain expertise in multiple
> frameworks, increasing complexity and reducing cross-team supportability.
> 2.
>
> *Inconsistent Feature Definitions:*
> - Transformations are implemented in different languages (PySpark
> for DE, Pandas for ML), which causes *feature drift*—where the
> same metrics or features calculated in separate environments diverge over
> time, leading to inconsistent results.
> 3.
>
> *Performance Bottlenecks in Production:*
> - While PySpark can regenerate features for serving models via
> APIs, it introduces inefficiencies:
> - *Overhead for Small Payloads:* Spark's distributed
> processing is unnecessary for small, real-time requests and is slower
> compared to Pandas.
> - *Latency Issues:* Spark struggles to deliver
> millisecond-level responses required in production APIs.
> - *Operational Complexity:* Maintaining Spark within API
> environments adds unnecessary overhead.
> 4.
>
> *Data Refinement Gap:*
> - Since ML teams are often formed after data engineering teams,
> they lag behind in leveraging years of PySpark-based data refinements.
> Reproducing the same transformations in Pandas to match the DE pipelines is
> time-consuming and error-prone.
>
>
> *The Ideal Scenario*
>
> To bridge the gap between Data Engineering and Machine Learning teams, the
> ideal solution would:
>
> 1.
>
> *Unify the Tech Stack:*
> - Both Data Engineers and ML teams would use *PySpark syntax* for data
> transformations and feature engineering. This shared language would
> simplify code maintenance, improve collaboration, and reduce the need for
> cross-training in multiple frameworks.
> 2.
>
> *Flexible Execution Backend:*
> - While using the same PySpark code, teams could choose the most
> efficient execution engine based on their needs:
> - *Data Engineers* would continue leveraging Spark's distributed
> processing for large-scale data transformations.
> - *ML Teams* could run the same PySpark transformations using
> *Pandas* as the processing engine for faster, on-the-fly feature
> generation in model training and API serving.
>
> This unified approach would eliminate redundant codebases, ensure
> consistent feature definitions, and optimize performance across both batch
> and real-time workflows.
>
>
> *The Proposal:*
>
> *Introduce an API that allows PySpark syntax while processing DataFrame
> using either Spark or Pandas depending on the session context.*
>
>
> *Simple, but intuitive example:*
>
> import pyspark.sql.functions as F
>
> def silver(bronze_df):
> return (
> bronze_df
> .withColumnRenamed("bronze_col", "silver_col")
> )
>
> def gold(silver_df):
> return (
> silver_df
> .withColumnRenamed("silver_col", "gold_col")
> .withColumn("gold_col", F.col("gold_col") + 1)
> )
>
> def features(gold_df):
> return (
> gold_df
> .withColumnRenamed("gold_col", "feature_col")
> .withColumn("feature_col", F.col("feature_col") + 1)
> )
>
> # With the Spark Session (normal way of using PySpark)
> spark = SparkSession.builder.master("local[1]").getOrCreate()
> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
> silver_df = silver(bronze_df)
> gold_df = gold(silver_df)
> features_df = features(gold_df)
> features_df.show()
>
> # Proposed "Pandas Spark Session"
> spark = SparkSession.builder.as_pandas.getOrCreate()
> # This would execute the same transformations using Pandas under the hood.
>
> This would enable teams to share the same codebase while choosing the most
> efficient processing engine.
>
> We've built and experimented in different data teams with a public
> library, *flyipe
> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
> that uses pandas_api() transformations, but can run using Pandas
> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
> but it still requires ML teams to manage separate pipelines for Spark
> dependencies.
>
> I’d love to hear thoughts from the community on this idea, and *if
> there's a better approach to solving this issue*.
>
> Thanks,
> José Müller
>
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Sean Owen <sr...@gmail.com>.
I don't think this makes sense, or lacks motivation. You want teams to
convert pandas code to Pyspark syntax, only to run it on pandas? why? just
run the pandas code in a larger job that also uses Spark if you like, or
within UDFs.
If you remove this assumption that people need to convert to Pyspark APIs,
then you're also left with pandas-on-Spark as an option. Or, just, pandas.
On Mon, Feb 10, 2025 at 6:02 AM José Müller <jo...@gmail.com>
wrote:
> Hi all,
>
> I'm new to the Spark community—please let me know if this isn’t the right
> forum for feature proposals.
>
> *About Me:*
> I’ve spent over 10 years in data roles, from engineering to machine
> learning and analytics. A recurring challenge I've encountered is the
> disconnect between data engineering and ML teams when it comes to feature
> engineering and data transformations using Spark.
>
> *The Problem:*
>
> There is a growing disconnect between Data Engineering and Machine
> Learning teams due to differences in the tools and languages they use for
> data transformations.
>
> -
>
> *Tooling Divergence:*
> - *Data Engineering Teams* primarily use *Spark SQL* and *PySpark* for
> building scalable data pipelines and transforming data across different
> layers (bronze, silver, gold).
> - *Machine Learning Teams* typically rely on *Pandas* for feature
> engineering, model training, and recreating features on-the-fly in
> production environments.
> -
>
> *Challenges Arising from This Divergence:*
> 1.
>
> *Code Duplication & Maintenance Overhead:*
> - ML teams often need to rebuild feature pipelines from raw data or
> adapt data engineer outputs into Pandas, leading to duplicated effort and
> maintenance in two different codebases.
> - This forces teams to maintain expertise in multiple
> frameworks, increasing complexity and reducing cross-team supportability.
> 2.
>
> *Inconsistent Feature Definitions:*
> - Transformations are implemented in different languages (PySpark
> for DE, Pandas for ML), which causes *feature drift*—where the
> same metrics or features calculated in separate environments diverge over
> time, leading to inconsistent results.
> 3.
>
> *Performance Bottlenecks in Production:*
> - While PySpark can regenerate features for serving models via
> APIs, it introduces inefficiencies:
> - *Overhead for Small Payloads:* Spark's distributed
> processing is unnecessary for small, real-time requests and is slower
> compared to Pandas.
> - *Latency Issues:* Spark struggles to deliver
> millisecond-level responses required in production APIs.
> - *Operational Complexity:* Maintaining Spark within API
> environments adds unnecessary overhead.
> 4.
>
> *Data Refinement Gap:*
> - Since ML teams are often formed after data engineering teams,
> they lag behind in leveraging years of PySpark-based data refinements.
> Reproducing the same transformations in Pandas to match the DE pipelines is
> time-consuming and error-prone.
>
>
> *The Ideal Scenario*
>
> To bridge the gap between Data Engineering and Machine Learning teams, the
> ideal solution would:
>
> 1.
>
> *Unify the Tech Stack:*
> - Both Data Engineers and ML teams would use *PySpark syntax* for data
> transformations and feature engineering. This shared language would
> simplify code maintenance, improve collaboration, and reduce the need for
> cross-training in multiple frameworks.
> 2.
>
> *Flexible Execution Backend:*
> - While using the same PySpark code, teams could choose the most
> efficient execution engine based on their needs:
> - *Data Engineers* would continue leveraging Spark's distributed
> processing for large-scale data transformations.
> - *ML Teams* could run the same PySpark transformations using
> *Pandas* as the processing engine for faster, on-the-fly feature
> generation in model training and API serving.
>
> This unified approach would eliminate redundant codebases, ensure
> consistent feature definitions, and optimize performance across both batch
> and real-time workflows.
>
>
> *The Proposal:*
>
> *Introduce an API that allows PySpark syntax while processing DataFrame
> using either Spark or Pandas depending on the session context.*
>
>
> *Simple, but intuitive example:*
>
> import pyspark.sql.functions as F
>
> def silver(bronze_df):
> return (
> bronze_df
> .withColumnRenamed("bronze_col", "silver_col")
> )
>
> def gold(silver_df):
> return (
> silver_df
> .withColumnRenamed("silver_col", "gold_col")
> .withColumn("gold_col", F.col("gold_col") + 1)
> )
>
> def features(gold_df):
> return (
> gold_df
> .withColumnRenamed("gold_col", "feature_col")
> .withColumn("feature_col", F.col("feature_col") + 1)
> )
>
> # With the Spark Session (normal way of using PySpark)
> spark = SparkSession.builder.master("local[1]").getOrCreate()
> bronze_df = spark.createDataFrame(schema=("bronze_col",), data=[(1,)])
> silver_df = silver(bronze_df)
> gold_df = gold(silver_df)
> features_df = features(gold_df)
> features_df.show()
>
> # Proposed "Pandas Spark Session"
> spark = SparkSession.builder.as_pandas.getOrCreate()
> # This would execute the same transformations using Pandas under the hood.
>
> This would enable teams to share the same codebase while choosing the most
> efficient processing engine.
>
> We've built and experimented in different data teams with a public
> library, *flyipe
> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
> that uses pandas_api() transformations, but can run using Pandas
> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
> but it still requires ML teams to manage separate pipelines for Spark
> dependencies.
>
> I’d love to hear thoughts from the community on this idea, and *if
> there's a better approach to solving this issue*.
>
> Thanks,
> José Müller
>
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Khalid Mammadov <kh...@gmail.com>.
There's also PolarSpark that offers PySpark API with Polars engine
https://github.com/khalidmammadov/polarspark
PS. It's also in PyPi
On Wed, 12 Feb 2025, 19:56 Sem, <ss...@apache.org> wrote:
> DuckDB provides "PySpark syntax" on top of fast single node engine:
>
> https://duckdb.org/docs/clients/python/spark_api.html
>
> As I remember, DuckDB is much faster than pandas on a single node and it
> already provides a spark-compatible API.
>
> On 2/10/25 1:02 PM, José Müller wrote:
> >
> > Hi all,
> >
> > I'm new to the Spark community—please let me know if this isn’t the
> > right forum for feature proposals.
> >
> > *About Me:*
> > I’ve spent over 10 years in data roles, from engineering to machine
> > learning and analytics. A recurring challenge I've encountered is the
> > disconnect between data engineering and ML teams when it comes to
> > feature engineering and data transformations using Spark.
> >
> > *The Problem:*
> >
> > There is a growing disconnect between Data Engineering and Machine
> > Learning teams due to differences in the tools and languages they use
> > for data transformations.
> >
> > *
> >
> > *Tooling Divergence:*
> >
> > o *Data Engineering Teams* primarily use *Spark SQL* and
> > *PySpark* for building scalable data pipelines and
> > transforming data across different layers (bronze, silver, gold).
> > o *Machine Learning Teams* typically rely on *Pandas* for
> > feature engineering, model training, and recreating features
> > on-the-fly in production environments.
> > *
> >
> > *Challenges Arising from This Divergence:*
> >
> > 1.
> >
> > *Code Duplication & Maintenance Overhead:*
> >
> > o ML teams often need to rebuild feature pipelines from raw
> > data or adapt data engineer outputs into Pandas, leading
> > to duplicated effort and maintenance in two different
> > codebases.
> > o This forces teams to maintain expertise in multiple
> > frameworks, increasing complexity and reducing cross-team
> > supportability.
> > 2.
> >
> > *Inconsistent Feature Definitions:*
> >
> > o Transformations are implemented in different languages
> > (PySpark for DE, Pandas for ML), which causes *feature
> > drift*—where the same metrics or features calculated in
> > separate environments diverge over time, leading to
> > inconsistent results.
> > 3.
> >
> > *Performance Bottlenecks in Production:*
> >
> > o While PySpark can regenerate features for serving models
> > via APIs, it introduces inefficiencies:
> > + *Overhead for Small Payloads:* Spark's distributed
> > processing is unnecessary for small, real-time
> > requests and is slower compared to Pandas.
> > + *Latency Issues:* Spark struggles to deliver
> > millisecond-level responses required in production APIs.
> > + *Operational Complexity:* Maintaining Spark within API
> > environments adds unnecessary overhead.
> > 4.
> >
> > *Data Refinement Gap:*
> >
> > o Since ML teams are often formed after data engineering
> > teams, they lag behind in leveraging years of
> > PySpark-based data refinements. Reproducing the same
> > transformations in Pandas to match the DE pipelines is
> > time-consuming and error-prone.
> >
> > *
> > *
> > *The Ideal Scenario*
> >
> > To bridge the gap between Data Engineering and Machine Learning teams,
> > the ideal solution would:
> >
> > 1.
> >
> > *Unify the Tech Stack:*
> >
> > * Both Data Engineers and ML teams would use *PySpark
> > syntax* for data transformations and feature engineering. This
> > shared language would simplify code maintenance, improve
> > collaboration, and reduce the need for cross-training in
> > multiple frameworks.
> > 2.
> >
> > *Flexible Execution Backend:*
> >
> > * While using the same PySpark code, teams could choose the most
> > efficient execution engine based on their needs:
> > o *Data Engineers* would continue leveraging Spark's
> > distributed processing for large-scale data transformations.
> > o *ML Teams* could run the same PySpark transformations
> > using *Pandas* as the processing engine for faster,
> > on-the-fly feature generation in model training and API
> > serving.
> >
> > This unified approach would eliminate redundant codebases, ensure
> > consistent feature definitions, and optimize performance across both
> > batch and real-time workflows.
> >
> >
> > *The Proposal:*
> >
> > *Introduce an API that allows PySpark syntax while processing
> > DataFrame using either Spark or Pandas depending on the session context.*
> >
> > *
> > *
> >
> > *Simple, but intuitive example:*
> >
> > import pyspark.sql.functionsas F
> >
> > def silver(bronze_df):
> > return (
> > bronze_df
> > .withColumnRenamed("bronze_col","silver_col")
> > )
> >
> > def gold(silver_df):
> > return (
> > silver_df
> > .withColumnRenamed("silver_col","gold_col")
> > .withColumn("gold_col", F.col("gold_col") +1)
> > )
> >
> > def features(gold_df):
> > return (
> > gold_df
> > .withColumnRenamed("gold_col","feature_col")
> > .withColumn("feature_col", F.col("feature_col") +1)
> > )
> >
> > # With the Spark Session (normal way of using PySpark) spark =
> SparkSession.builder.master("local[1]").getOrCreate()
> > bronze_df = spark.createDataFrame(schema=("bronze_col",),data=[(1,)])
> > silver_df = silver(bronze_df)
> > gold_df = gold(silver_df)
> > features_df = features(gold_df)
> > features_df.show()
> >
> > # Proposed "Pandas Spark Session" spark =
> SparkSession.builder.as_pandas.getOrCreate()
> > # This would execute the same transformations using Pandas under the
> hood.
> >
> > This would enable teams to share the same codebase while choosing the
> > most efficient processing engine.
> >
> > We've built and experimented in different data teams with a public
> > library, *flyipe
> > <
> https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
>
> > that uses |pandas_api()| transformations, but can run using Pandas
> > <
> https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
>
> > but it still requires ML teams to manage separate pipelines for Spark
> > dependencies.
> >
> > I’d love to hear thoughts from the community on this idea, and *if
> > there's a better approach to solving this issue*.
> >
> > Thanks,
> > José Müller
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [PROPOSAL] Unified PySpark-Pandas API to Bridge Data Engineering and ML Workflows
Posted by Sem <ss...@apache.org>.
DuckDB provides "PySpark syntax" on top of fast single node engine:
https://duckdb.org/docs/clients/python/spark_api.html
As I remember, DuckDB is much faster than pandas on a single node and it
already provides a spark-compatible API.
On 2/10/25 1:02 PM, José Müller wrote:
>
> Hi all,
>
> I'm new to the Spark community—please let me know if this isn’t the
> right forum for feature proposals.
>
> *About Me:*
> I’ve spent over 10 years in data roles, from engineering to machine
> learning and analytics. A recurring challenge I've encountered is the
> disconnect between data engineering and ML teams when it comes to
> feature engineering and data transformations using Spark.
>
> *The Problem:*
>
> There is a growing disconnect between Data Engineering and Machine
> Learning teams due to differences in the tools and languages they use
> for data transformations.
>
> *
>
> *Tooling Divergence:*
>
> o *Data Engineering Teams* primarily use *Spark SQL* and
> *PySpark* for building scalable data pipelines and
> transforming data across different layers (bronze, silver, gold).
> o *Machine Learning Teams* typically rely on *Pandas* for
> feature engineering, model training, and recreating features
> on-the-fly in production environments.
> *
>
> *Challenges Arising from This Divergence:*
>
> 1.
>
> *Code Duplication & Maintenance Overhead:*
>
> o ML teams often need to rebuild feature pipelines from raw
> data or adapt data engineer outputs into Pandas, leading
> to duplicated effort and maintenance in two different
> codebases.
> o This forces teams to maintain expertise in multiple
> frameworks, increasing complexity and reducing cross-team
> supportability.
> 2.
>
> *Inconsistent Feature Definitions:*
>
> o Transformations are implemented in different languages
> (PySpark for DE, Pandas for ML), which causes *feature
> drift*—where the same metrics or features calculated in
> separate environments diverge over time, leading to
> inconsistent results.
> 3.
>
> *Performance Bottlenecks in Production:*
>
> o While PySpark can regenerate features for serving models
> via APIs, it introduces inefficiencies:
> + *Overhead for Small Payloads:* Spark's distributed
> processing is unnecessary for small, real-time
> requests and is slower compared to Pandas.
> + *Latency Issues:* Spark struggles to deliver
> millisecond-level responses required in production APIs.
> + *Operational Complexity:* Maintaining Spark within API
> environments adds unnecessary overhead.
> 4.
>
> *Data Refinement Gap:*
>
> o Since ML teams are often formed after data engineering
> teams, they lag behind in leveraging years of
> PySpark-based data refinements. Reproducing the same
> transformations in Pandas to match the DE pipelines is
> time-consuming and error-prone.
>
> *
> *
> *The Ideal Scenario*
>
> To bridge the gap between Data Engineering and Machine Learning teams,
> the ideal solution would:
>
> 1.
>
> *Unify the Tech Stack:*
>
> * Both Data Engineers and ML teams would use *PySpark
> syntax* for data transformations and feature engineering. This
> shared language would simplify code maintenance, improve
> collaboration, and reduce the need for cross-training in
> multiple frameworks.
> 2.
>
> *Flexible Execution Backend:*
>
> * While using the same PySpark code, teams could choose the most
> efficient execution engine based on their needs:
> o *Data Engineers* would continue leveraging Spark's
> distributed processing for large-scale data transformations.
> o *ML Teams* could run the same PySpark transformations
> using *Pandas* as the processing engine for faster,
> on-the-fly feature generation in model training and API
> serving.
>
> This unified approach would eliminate redundant codebases, ensure
> consistent feature definitions, and optimize performance across both
> batch and real-time workflows.
>
>
> *The Proposal:*
>
> *Introduce an API that allows PySpark syntax while processing
> DataFrame using either Spark or Pandas depending on the session context.*
>
> *
> *
>
> *Simple, but intuitive example:*
>
> import pyspark.sql.functionsas F
>
> def silver(bronze_df):
> return (
> bronze_df
> .withColumnRenamed("bronze_col","silver_col")
> )
>
> def gold(silver_df):
> return (
> silver_df
> .withColumnRenamed("silver_col","gold_col")
> .withColumn("gold_col", F.col("gold_col") +1)
> )
>
> def features(gold_df):
> return (
> gold_df
> .withColumnRenamed("gold_col","feature_col")
> .withColumn("feature_col", F.col("feature_col") +1)
> )
>
> # With the Spark Session (normal way of using PySpark) spark = SparkSession.builder.master("local[1]").getOrCreate()
> bronze_df = spark.createDataFrame(schema=("bronze_col",),data=[(1,)])
> silver_df = silver(bronze_df)
> gold_df = gold(silver_df)
> features_df = features(gold_df)
> features_df.show()
>
> # Proposed "Pandas Spark Session" spark = SparkSession.builder.as_pandas.getOrCreate()
> # This would execute the same transformations using Pandas under the hood.
>
> This would enable teams to share the same codebase while choosing the
> most efficient processing engine.
>
> We've built and experimented in different data teams with a public
> library, *flyipe
> <https://flypipe.github.io/flypipe/html/release/4.1.0/index.html#what-flypipe-aims-to-facilitate>*,
> that uses |pandas_api()| transformations, but can run using Pandas
> <https://flypipe.github.io/flypipe/html/release/4.1.0/notebooks/tutorial/multiple-node-types.html#4.-pandas_on_spark-nodes-as-pandas>,
> but it still requires ML teams to manage separate pipelines for Spark
> dependencies.
>
> I’d love to hear thoughts from the community on this idea, and *if
> there's a better approach to solving this issue*.
>
> Thanks,
> José Müller
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org