Re: Use case idea

[pengyh <pe...@web.de>]

I am afraid the most sql functions spark has the other BI tools also have.

spark is used for high performance computing, not for SQL function
comparisoin.

Thanks.

> In other terms: what analytics funcionality, that no One erp has, Spark offers ?

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Use case idea

[pengyh <pe...@web.de>]

* streaming handler is still useful for spark, though there is flink as
alternative
* RDD is also useful for transform especially for non-structure data
* there are many SQL products in market like Drill/Impala, but spark is
more powerful for distributed deployment as far as I know
* we never used spark for AI training, but use keras/pytorch which are
pretty easy for development a model.

> Perhaps you should try other systems in the market first, that will give
> an unbiased view of databricks and SPARK being just over
> glamourised tool. The hope of extending SPARK with a separate easy to
> use query engine for deep learning and other AI systems is gone now with
> Ray, SPARK community now just defends the lack of support, and direction
> in this matter largely, which is a joke.


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Use case idea

[Gourav Sengupta <go...@gmail.com>]

Hi,

my comments were for purposes of SQL, also most of other technologies like
snowflake, and Redshift, and using KSQL directly to other sinks quite
easily, without massive engineering, infact databricks is trying to play a
catchup game in this market by coming out with GIU based ETL tools :)

Perhaps you should try other systems in the market first, that will give an
unbiased view of databricks and SPARK being just over glamourised tool. The
hope of extending SPARK with a separate easy to use query engine for deep
learning and other AI systems is gone now with Ray, SPARK community now
just defends the lack of support, and direction in this matter largely,
which is a joke.

Thanks and Regards,
Gourav Sengupta

On Mon, Aug 1, 2022 at 4:54 AM pengyh <pe...@web.de> wrote:

>
> I don't think so. we were using spark integarted with Kafka for
> streaming computing and realtime reports. that just works.
>
>
> > SPARK is now just an overhyped and overcomplicated ETL tool, nothing
> > more, there is another distributed AI called as Ray, which should be the
> > next billion dollar company instead of just building those features in
> > SPARK natively using a different computation engine :)
>

Re: Use case idea

[pengyh <pe...@web.de>]

I don't think so. we were using spark integarted with Kafka for
streaming computing and realtime reports. that just works.


> SPARK is now just an overhyped and overcomplicated ETL tool, nothing
> more, there is another distributed AI called as Ray, which should be the
> next billion dollar company instead of just building those features in
> SPARK natively using a different computation engine :)

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Use case idea

[Gourav Sengupta <go...@gmail.com>]

Hi,
SPARK is now just an overhyped and overcomplicated ETL tool, nothing more,
there is another distributed AI called as Ray, which should be the next
billion dollar company instead of just building those features in SPARK
natively using a different computation engine :)

So the only promise of SPARK growing into a truly unified platform is gone
now, who wants to build features into SPARK anymore when you can build
another billion dollar hype :)

Use solutions like snowflake, or BigQuery (may turn out to be expensive),
or EMR based hive/ trino, or Redshift, Redshift Spectrum, Athena, and so on.


Thanks and Regards,
Gourav Sengupta

On Mon, Aug 1, 2022 at 1:58 AM pengyh <pe...@web.de> wrote:

>
> I am afraid the most sql functions spark has the other BI tools also have.
>
> spark is used for high performance computing, not for SQL function
> comparisoin.
>
> Thanks.
>
> > In other terms: what analytics funcionality, that no One erp has, Spark
> offers ?
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>