[jira] [Updated] (SPARK-53317) [spark-connect-rust] Registry a DataFrame from Polars

["Steve Russo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-53317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Russo updated SPARK-53317:
--------------------------------
    Description: 
The rust client only supports the ability to `create_dataframe` from ArrowBatch. To increase the usability of the client an additional method should be create to allow for the creation of a dataframe from a polars dataframe.

Example provided below.

 
{code:java}
use polars::prelude::*;
use spark_connect_rs::{SparkSession, SparkSessionBuilder};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {     
    let spark: SparkSession = SparkSessionBuilder::remote("sc://127.0.0.1:15002/")
         .build()
         .await?;

    let s1 = Series::new("Name", &["Alice", "Bob", "Charlie"]);
    let s2 = Series::new("Age", &[30, 24, 35]);
    let s3 = Series::new("City", &["New York", "London", "Paris"]);
    let df: DataFrame = DataFrame::new(vec![s1, s2, s3])?;

    // from_polars    
    let spark_df = spark.from_polars(df)    

    spark_df.show(Some(4), None, None).await?;    
    Ok(()) 
} {code}
 

  was:
The rust client only supports the ability to `create_dataframe` from ArrowBatch. To increase the usability of the client an additional method should be create to allow for the creation of a dataframe from a polars dataframe.

Example provided below.

```rust

use polars::prelude::*;

use spark_connect_rs::\{SparkSession, SparkSessionBuilder};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {

    let spark: SparkSession = SparkSessionBuilder::remote("sc://127.0.0.1:15002/")
        .build()
        .await?;

 

    let s1 = Series::new("Name", &["Alice", "Bob", "Charlie"]);
    let s2 = Series::new("Age", &[30, 24, 35]);
    let s3 = Series::new("City", &["New York", "London", "Paris"]);

    let df: DataFrame = DataFrame::new(vec![s1, s2, s3])?;

 

   // from_polars

   let spark_df = spark.from_polars(df)


   spark_df.show(Some(4), None, None).await?;
   Ok(())

}

```


> [spark-connect-rust] Registry a DataFrame from Polars
> -----------------------------------------------------
>
>                 Key: SPARK-53317
>                 URL: https://issues.apache.org/jira/browse/SPARK-53317
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-0.1.0
>            Reporter: Steve Russo
>            Priority: Major
>
> The rust client only supports the ability to `create_dataframe` from ArrowBatch. To increase the usability of the client an additional method should be create to allow for the creation of a dataframe from a polars dataframe.
> Example provided below.
>  
> {code:java}
> use polars::prelude::*;
> use spark_connect_rs::{SparkSession, SparkSessionBuilder};
> #[tokio::main]
> async fn main() -> Result<(), Box<dyn std::error::Error>> {     
>     let spark: SparkSession = SparkSessionBuilder::remote("sc://127.0.0.1:15002/")
>          .build()
>          .await?;
>     let s1 = Series::new("Name", &["Alice", "Bob", "Charlie"]);
>     let s2 = Series::new("Age", &[30, 24, 35]);
>     let s3 = Series::new("City", &["New York", "London", "Paris"]);
>     let df: DataFrame = DataFrame::new(vec![s1, s2, s3])?;
>     // from_polars    
>     let spark_df = spark.from_polars(df)    
>     spark_df.show(Some(4), None, None).await?;    
>     Ok(()) 
> } {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org