SchemaRDD to RDD[String]

[Hafiz Mujadid <ha...@gmail.com>]

Hi dears!

I want to convert a schemaRDD into RDD of String. How can we do that?

Currently I am doing like this which is not converting correctly no
exception but resultant strings are empty


here is my code
def SchemaRDDToRDD( schemaRDD : SchemaRDD ) : RDD[ String ] = {
		var types = schemaRDD.schema.fields.map( field => field.dataType )
		var list = new ArrayList[ String ]()
		types.foreach( Type => {
			list.add( Type.toString )
		} )
		schemaRDD.map( row => rowToString( row, list ) )

	}

	private def rowToString( row : Row, list : ArrayList[ String ] ) : String =
{
		var record = new StringBuilder
		for ( i <- 0 until row.length ) {
			record.append( getValue( list.get( i ), row, i ) )
			record.append( "," )
		}
		var sub=record.setLength( record.length - 1 ).toString
		println(sub)
		sub
	}
	/** get a single value from row object
	 *  @param datatype of value
	 *  @param record of type Row
	 *  @param i index of column in row object
	 *  @return value in string
	 */
	private def getValue( dataType : String, record : Row, i : Int ) : String =
{
		var res=""
		if ( dataType.equalsIgnoreCase( "IntegerType" ) )
			res+=record.getInt( i )
		else if ( dataType.equalsIgnoreCase( "FloatType" ) )
			res+=record.getFloat( i )
		else if ( dataType.equalsIgnoreCase( "LongType" ) )
			res+=record.getLong( i )
		else if ( dataType.equalsIgnoreCase( "DoubleType" ) )
			res+=record.getDouble( i )
		else if ( dataType.equalsIgnoreCase( "TimestampType" ) )
			res+=record.getString( i )
		else if ( dataType.equalsIgnoreCase( "ShortType" ) )
			res+=record.getShort( i )
		else if ( dataType.equalsIgnoreCase( "ByteType" ) )
			res+=record.getByte( i )
		else if ( dataType.equalsIgnoreCase( "StringType" ) )
			res=record.getString( i )
		else
			res=record.getString( i )
		println(res)
		res
	}







--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SchemaRDD-to-RDD-String-tp20846.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: SchemaRDD to RDD[String]

[Yana <ya...@gmail.com>]

Do your debug println show values? i.e. what would you see if in rowToString
you output println(" row to string "+row+" "+sub)? 

Another thing to check would be to do  schemaRDD.take(3) or something to
make sure you actually have data....

you can also try this: rowToString(schemaRDD.first,list) and see if you get
anything



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SchemaRDD-to-RDD-String-tp20846p20910.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: SchemaRDD to RDD[String]

[Michael Armbrust <mi...@databricks.com>]

You might also try the following, which I think is equivalent:

schemaRDD.map(_.mkString(","))

On Wed, Dec 24, 2014 at 8:12 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:

> Hi,
>
> On Wed, Dec 24, 2014 at 3:18 PM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>>
>> I want to convert a schemaRDD into RDD of String. How can we do that?
>>
>> Currently I am doing like this which is not converting correctly no
>> exception but resultant strings are empty
>>
>> here is my code
>>
>
> Hehe, this is the most Java-ish Scala code I have ever seen ;-)
>
> Having said that, are you sure that your rows are not empty? The code
> looks correct to me, actually.
>
> Tobias
>
>
>

Re: SchemaRDD to RDD[String]

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

On Wed, Dec 24, 2014 at 3:18 PM, Hafiz Mujadid <ha...@gmail.com>
wrote:
>
> I want to convert a schemaRDD into RDD of String. How can we do that?
>
> Currently I am doing like this which is not converting correctly no
> exception but resultant strings are empty
>
> here is my code
>

Hehe, this is the most Java-ish Scala code I have ever seen ;-)

Having said that, are you sure that your rows are not empty? The code looks
correct to me, actually.

Tobias