You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jun Shi <ju...@gmail.com> on 2017/11/02 02:54:17 UTC
Fwd: Dose pyspark supports python3.6？
Dear spark developers：
It’s so exciting to send this email to you.
I have encountered the problem that if pyspark supports python3.6?
(I found some answer online is no.) Can you tell me the answer which
python versions does pyspark support?
I’m looking forward for your answer. Thank you very much!
Best,
Jun
Re: Fwd: Dose pyspark supports python3.6？
Posted by makoto <to...@gmail.com>.
I'm not sure whether pyspark supports python 3.6 but pyspark and python
3.6 is working on my environment.
I found the following issue and it seems to be already resolved.
https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-19019
2017/11/02 午前11:54 "Jun Shi" <ju...@gmail.com>:
Dear spark developers：
It’s so exciting to send this email to you.
I have encountered the problem that if pyspark supports python3.6?
(I found some answer online is no.) Can you tell me the answer which
python versions does pyspark support?
I’m looking forward for your answer. Thank you very much!
Best,
Jun
Spark as ETL, was: Re: Dose pyspark supports python3.6？
Posted by JG Perrin <jp...@lumeris.com>.
Pros:
No need for Scala skills, Java can be used.
Other companies are already doing it.
> Support Yarn execution
But not only…
Complex use-case for import can easily be done in Java (see https://spark-summit.org/eu-2017/events/extending-apache-sparks-ingestion-building-your-own-java-data-source/ - sorry shameless self promo).
Can be parallelized on all nodes of the cluster
Cons:
> No ETL gui.
But works very well with libraries like log4j so we can track the process
His
jg
On Nov 1, 2017, at 22:58, van den Heever, Christian CC <Ch...@standardbank.co.za>> wrote:
Dear Spark users
I have been asked to provide a presentation / business case as to why to use spark and java as ingestion tool for HDFS and HIVE
And why to move away from an etl tool.
Could you be so kind as to provide with some pros and cons to this.
I have the following :
Pros:
In house build – code can be changes on the fly to suite business need.
Software is free
Can out of the box run on all nodes
Will support all Apache based software.
Fast deu to in memory processing
Spark UI can visualise execution
Support checkpoint data loads
Support echama regesty for custom schema and inference.
Support Yarn execution
Mlibs can be used in need.
Data linage support deu to spar usage.
Cons
Skills needed to maintain and build
In memory cabibility can become bottleneck if not managed
No ETL gui.
Maybe point be to an article if you have one.
Thanks a mill.
Christian
Standard Bank email disclaimer and confidentiality note
Please go to www.standardbank.co.za/site/homepage/emaildisclaimer.html<http://www.standardbank.co.za/site/homepage/emaildisclaimer.html> to read our email disclaimer and confidentiality note. Kindly email disclaimer@standardbank.co.za<ma...@standardbank.co.za> (no content or subject line necessary) if you cannot view that page and we will email our email disclaimer and confidentiality note to you.
RE: Dose pyspark supports python3.6？
Posted by "van den Heever, Christian CC" <Ch...@standardbank.co.za>.
Dear Spark users
I have been asked to provide a presentation / business case as to why to use spark and java as ingestion tool for HDFS and HIVE
And why to move away from an etl tool.
Could you be so kind as to provide with some pros and cons to this.
I have the following :
Pros:
In house build – code can be changes on the fly to suite business need.
Software is free
Can out of the box run on all nodes
Will support all Apache based software.
Fast deu to in memory processing
Spark UI can visualise execution
Support checkpoint data loads
Support echama regesty for custom schema and inference.
Support Yarn execution
Mlibs can be used in need.
Data linage support deu to spar usage.
Cons
Skills needed to maintain and build
In memory cabibility can become bottleneck if not managed
No ETL gui.
Maybe point be to an article if you have one.
Thanks a mill.
Christian
Standard Bank email disclaimer and confidentiality note
Please go to www.standardbank.co.za/site/homepage/emaildisclaimer.html to read our email disclaimer and confidentiality note. Kindly email disclaimer@standardbank.co.za (no content or subject line necessary) if you cannot view that page and we will email our email disclaimer and confidentiality note to you.