You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by bo yang <bo...@gmail.com> on 2023/09/12 16:36:01 UTC
Write Spark Connection client application in Go
Hi Spark Friends,
Anyone interested in using Golang to write Spark application? We
created a Spark
Connect Go Client library <https://github.com/apache/spark-connect-go>.
Would love to hear feedback/thoughts from the community.
Please see the quick start guide
<https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
about how to use it. Following is a very short Spark Connect application in
Go:
func main() {
spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
defer spark.Stop()
df, _ := spark.Sql("select 'apple' as word, 123 as count union all
select 'orange' as word, 456 as count")
df.Show(100, false)
df.Collect()
df.Write().Mode("overwrite").
Format("parquet").
Save("file:///tmp/spark-connect-write-example-output.parquet")
df = spark.Read().Format("parquet").
Load("file:///tmp/spark-connect-write-example-output.parquet")
df.Show(100, false)
df.CreateTempView("view1", true, false)
df, _ = spark.Sql("select count, word from view1 order by count")
}
Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and working
together on this repo! Welcome more people to contribute :)
Best,
Bo
Re: Write Spark Connection client application in Go
Posted by bo yang <bo...@gmail.com>.
Thanks Holden and Martin for the nice words and feedback :)
On Wed, Sep 13, 2023 at 8:22 AM Martin Grund <ma...@databricks.com> wrote:
> This is absolutely awesome! Thank you so much for dedicating your time to
> this project!
>
>
> On Wed, Sep 13, 2023 at 6:04 AM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> That’s so cool! Great work y’all :)
>>
>> On Tue, Sep 12, 2023 at 8:14 PM bo yang <bo...@gmail.com> wrote:
>>
>>> Hi Spark Friends,
>>>
>>> Anyone interested in using Golang to write Spark application? We created
>>> a Spark Connect Go Client library
>>> <https://github.com/apache/spark-connect-go>. Would love to hear
>>> feedback/thoughts from the community.
>>>
>>> Please see the quick start guide
>>> <https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
>>> about how to use it. Following is a very short Spark Connect application in
>>> Go:
>>>
>>> func main() {
>>> spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
>>> defer spark.Stop()
>>>
>>> df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
>>> df.Show(100, false)
>>> df.Collect()
>>>
>>> df.Write().Mode("overwrite").
>>> Format("parquet").
>>> Save("file:///tmp/spark-connect-write-example-output.parquet")
>>>
>>> df = spark.Read().Format("parquet").
>>> Load("file:///tmp/spark-connect-write-example-output.parquet")
>>> df.Show(100, false)
>>>
>>> df.CreateTempView("view1", true, false)
>>> df, _ = spark.Sql("select count, word from view1 order by count")
>>> }
>>>
>>>
>>> Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and
>>> working together on this repo! Welcome more people to contribute :)
>>>
>>> Best,
>>> Bo
>>>
>>>
Re: Write Spark Connection client application in Go
Posted by bo yang <bo...@gmail.com>.
Thanks Holden and Martin for the nice words and feedback :)
On Wed, Sep 13, 2023 at 8:22 AM Martin Grund <ma...@databricks.com> wrote:
> This is absolutely awesome! Thank you so much for dedicating your time to
> this project!
>
>
> On Wed, Sep 13, 2023 at 6:04 AM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> That’s so cool! Great work y’all :)
>>
>> On Tue, Sep 12, 2023 at 8:14 PM bo yang <bo...@gmail.com> wrote:
>>
>>> Hi Spark Friends,
>>>
>>> Anyone interested in using Golang to write Spark application? We created
>>> a Spark Connect Go Client library
>>> <https://github.com/apache/spark-connect-go>. Would love to hear
>>> feedback/thoughts from the community.
>>>
>>> Please see the quick start guide
>>> <https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
>>> about how to use it. Following is a very short Spark Connect application in
>>> Go:
>>>
>>> func main() {
>>> spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
>>> defer spark.Stop()
>>>
>>> df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
>>> df.Show(100, false)
>>> df.Collect()
>>>
>>> df.Write().Mode("overwrite").
>>> Format("parquet").
>>> Save("file:///tmp/spark-connect-write-example-output.parquet")
>>>
>>> df = spark.Read().Format("parquet").
>>> Load("file:///tmp/spark-connect-write-example-output.parquet")
>>> df.Show(100, false)
>>>
>>> df.CreateTempView("view1", true, false)
>>> df, _ = spark.Sql("select count, word from view1 order by count")
>>> }
>>>
>>>
>>> Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and
>>> working together on this repo! Welcome more people to contribute :)
>>>
>>> Best,
>>> Bo
>>>
>>>
Re: Write Spark Connection client application in Go
Posted by Martin Grund <ma...@databricks.com.INVALID>.
This is absolutely awesome! Thank you so much for dedicating your time to
this project!
On Wed, Sep 13, 2023 at 6:04 AM Holden Karau <ho...@pigscanfly.ca> wrote:
> That’s so cool! Great work y’all :)
>
> On Tue, Sep 12, 2023 at 8:14 PM bo yang <bo...@gmail.com> wrote:
>
>> Hi Spark Friends,
>>
>> Anyone interested in using Golang to write Spark application? We created
>> a Spark Connect Go Client library
>> <https://github.com/apache/spark-connect-go>. Would love to hear
>> feedback/thoughts from the community.
>>
>> Please see the quick start guide
>> <https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
>> about how to use it. Following is a very short Spark Connect application in
>> Go:
>>
>> func main() {
>> spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
>> defer spark.Stop()
>>
>> df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
>> df.Show(100, false)
>> df.Collect()
>>
>> df.Write().Mode("overwrite").
>> Format("parquet").
>> Save("file:///tmp/spark-connect-write-example-output.parquet")
>>
>> df = spark.Read().Format("parquet").
>> Load("file:///tmp/spark-connect-write-example-output.parquet")
>> df.Show(100, false)
>>
>> df.CreateTempView("view1", true, false)
>> df, _ = spark.Sql("select count, word from view1 order by count")
>> }
>>
>>
>> Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and
>> working together on this repo! Welcome more people to contribute :)
>>
>> Best,
>> Bo
>>
>>
Re: Write Spark Connection client application in Go
Posted by Martin Grund <ma...@databricks.com.INVALID>.
This is absolutely awesome! Thank you so much for dedicating your time to
this project!
On Wed, Sep 13, 2023 at 6:04 AM Holden Karau <ho...@pigscanfly.ca> wrote:
> That’s so cool! Great work y’all :)
>
> On Tue, Sep 12, 2023 at 8:14 PM bo yang <bo...@gmail.com> wrote:
>
>> Hi Spark Friends,
>>
>> Anyone interested in using Golang to write Spark application? We created
>> a Spark Connect Go Client library
>> <https://github.com/apache/spark-connect-go>. Would love to hear
>> feedback/thoughts from the community.
>>
>> Please see the quick start guide
>> <https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
>> about how to use it. Following is a very short Spark Connect application in
>> Go:
>>
>> func main() {
>> spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
>> defer spark.Stop()
>>
>> df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
>> df.Show(100, false)
>> df.Collect()
>>
>> df.Write().Mode("overwrite").
>> Format("parquet").
>> Save("file:///tmp/spark-connect-write-example-output.parquet")
>>
>> df = spark.Read().Format("parquet").
>> Load("file:///tmp/spark-connect-write-example-output.parquet")
>> df.Show(100, false)
>>
>> df.CreateTempView("view1", true, false)
>> df, _ = spark.Sql("select count, word from view1 order by count")
>> }
>>
>>
>> Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and
>> working together on this repo! Welcome more people to contribute :)
>>
>> Best,
>> Bo
>>
>>
Re: Write Spark Connection client application in Go
Posted by Holden Karau <ho...@pigscanfly.ca>.
That’s so cool! Great work y’all :)
On Tue, Sep 12, 2023 at 8:14 PM bo yang <bo...@gmail.com> wrote:
> Hi Spark Friends,
>
> Anyone interested in using Golang to write Spark application? We created a Spark
> Connect Go Client library <https://github.com/apache/spark-connect-go>.
> Would love to hear feedback/thoughts from the community.
>
> Please see the quick start guide
> <https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
> about how to use it. Following is a very short Spark Connect application in
> Go:
>
> func main() {
> spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
> defer spark.Stop()
>
> df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
> df.Show(100, false)
> df.Collect()
>
> df.Write().Mode("overwrite").
> Format("parquet").
> Save("file:///tmp/spark-connect-write-example-output.parquet")
>
> df = spark.Read().Format("parquet").
> Load("file:///tmp/spark-connect-write-example-output.parquet")
> df.Show(100, false)
>
> df.CreateTempView("view1", true, false)
> df, _ = spark.Sql("select count, word from view1 order by count")
> }
>
>
> Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and working
> together on this repo! Welcome more people to contribute :)
>
> Best,
> Bo
>
>
Re: Write Spark Connection client application in Go
Posted by Holden Karau <ho...@pigscanfly.ca>.
That’s so cool! Great work y’all :)
On Tue, Sep 12, 2023 at 8:14 PM bo yang <bo...@gmail.com> wrote:
> Hi Spark Friends,
>
> Anyone interested in using Golang to write Spark application? We created a Spark
> Connect Go Client library <https://github.com/apache/spark-connect-go>.
> Would love to hear feedback/thoughts from the community.
>
> Please see the quick start guide
> <https://github.com/apache/spark-connect-go/blob/master/quick-start.md>
> about how to use it. Following is a very short Spark Connect application in
> Go:
>
> func main() {
> spark, _ := sql.SparkSession.Builder.Remote("sc://localhost:15002").Build()
> defer spark.Stop()
>
> df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
> df.Show(100, false)
> df.Collect()
>
> df.Write().Mode("overwrite").
> Format("parquet").
> Save("file:///tmp/spark-connect-write-example-output.parquet")
>
> df = spark.Read().Format("parquet").
> Load("file:///tmp/spark-connect-write-example-output.parquet")
> df.Show(100, false)
>
> df.CreateTempView("view1", true, false)
> df, _ = spark.Sql("select count, word from view1 order by count")
> }
>
>
> Many thanks to Martin, Hyukjin, Ruifeng and Denny for creating and working
> together on this repo! Welcome more people to contribute :)
>
> Best,
> Bo
>
>