You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Franc Carter <fr...@rozettatech.com> on 2015/08/19 07:48:00 UTC
SparkR csv without headers
Hi,
Does anyone have an example of how to create a DataFrame in SparkR which
specifies the column names - the csv files I have do not have column names
in the first row. I can get read a csv nicely with
com.databricks:spark-csv_2.10:1.0.3, but I end up with column names C1, C2,
C3 etc
thanks
--
*Franc Carter* I Systems Architect I RoZetta Technology
[image: Description: Description: Description:
cid:image003.jpg@01D02903.9B540580]
L4. 55 Harrington Street, THE ROCKS, NSW, 2000
PO Box H58, Australia Square, Sydney NSW, 1215, AUSTRALIA
*T* +61 2 8355 2515 I www.rozettatechnology.com
[image: cid:image002.jpg@01D02903.0B41B280]
DISCLAIMER: The contents of this email, inclusive of attachments, may be
legally
privileged and confidential. Any unauthorised use of the contents is
expressly prohibited.
Re: SparkR csv without headers
Posted by Franc Carter <fr...@rozettatech.com>.
Thanks - works nicely
cheers
On Fri, Aug 21, 2015 at 12:43 PM, Sun, Rui <ru...@intel.com> wrote:
> Hi,
>
>
>
> You can create a DataFrame using load.df() with a specified schema.
>
>
>
> Something like:
>
> schema <- structType(structField(“a”, “string”), structField(“b”,
> integer), …)
>
> read.df ( …, schema = schema)
>
>
>
> *From:* Franc Carter [mailto:franc.carter@rozettatech.com]
> *Sent:* Wednesday, August 19, 2015 1:48 PM
> *To:* user@spark.apache.org
> *Subject:* SparkR csv without headers
>
>
>
>
>
> Hi,
>
>
>
> Does anyone have an example of how to create a DataFrame in SparkR which
> specifies the column names - the csv files I have do not have column names
> in the first row. I can get read a csv nicely with
> com.databricks:spark-csv_2.10:1.0.3, but I end up with column names C1, C2,
> C3 etc
>
>
>
>
>
> thanks
>
>
>
> --
>
> *Franc Carter* I Systems Architect I RoZetta Technology
>
>
>
> [image: Description: Description: Description:
> cid:image003.jpg@01D02903.9B540580]
>
>
>
> L4. 55 Harrington Street, THE ROCKS, NSW, 2000
>
> PO Box H58, Australia Square, Sydney NSW, 1215, AUSTRALIA
>
> *T* +61 2 8355 2515 I www.rozettatechnology.com
>
> [image: cid:image002.jpg@01D02903.0B41B280]
>
> DISCLAIMER: The contents of this email, inclusive of attachments, may be
> legally
>
> privileged and confidential. Any unauthorised use of the contents is
> expressly prohibited.
>
>
>
>
>
--
*Franc Carter* I Systems Architect I RoZetta Technology
[image: Description: Description: Description:
cid:image003.jpg@01D02903.9B540580]
L4. 55 Harrington Street, THE ROCKS, NSW, 2000
PO Box H58, Australia Square, Sydney NSW, 1215, AUSTRALIA
*T* +61 2 8355 2515 I www.rozettatechnology.com
[image: cid:image002.jpg@01D02903.0B41B280]
DISCLAIMER: The contents of this email, inclusive of attachments, may be
legally
privileged and confidential. Any unauthorised use of the contents is
expressly prohibited.
RE: SparkR csv without headers
Posted by Felix Cheung <fe...@hotmail.com>.
You could also rename them with names
Unfortunately the API doesn't show the example of that https://spark.apache.org/docs/latest/api/R/index.html
On Thu, Aug 20, 2015 at 7:43 PM -0700, "Sun, Rui" <ru...@intel.com> wrote:
Hi,
You can create a DataFrame using load.df() with a specified schema.
Something like:
schema <- structType(structField(“a”, “string”), structField(“b”, integer), …)
read.df ( …, schema = schema)
From: Franc Carter [mailto:franc.carter@rozettatech.com]
Sent: Wednesday, August 19, 2015 1:48 PM
To: user@spark.apache.org
Subject: SparkR csv without headers
Hi,
Does anyone have an example of how to create a DataFrame in SparkR which specifies the column names - the csv files I have do not have column names in the first row. I can get read a csv nicely with com.databricks:spark-csv_2.10:1.0.3, but I end up with column names C1, C2, C3 etc
thanks
--
Franc Carter I Systems Architect I RoZetta Technology
[Description: Description: Description: cid:image003.jpg@01D02903.9B540580]
L4. 55 Harrington Street, THE ROCKS, NSW, 2000
PO Box H58, Australia Square, Sydney NSW, 1215, AUSTRALIA
T +61 2 8355 2515<tel:%2B61%202%208355%202515> I www.rozettatechnology.com<http://www.rozettatechnology.com/>
[cid:image002.jpg@01D02903.0B41B280]
DISCLAIMER: The contents of this email, inclusive of attachments, may be legally
privileged and confidential. Any unauthorised use of the contents is expressly prohibited.
RE: SparkR csv without headers
Posted by "Sun, Rui" <ru...@intel.com>.
Hi,
You can create a DataFrame using load.df() with a specified schema.
Something like:
schema <- structType(structField(“a”, “string”), structField(“b”, integer), …)
read.df ( …, schema = schema)
From: Franc Carter [mailto:franc.carter@rozettatech.com]
Sent: Wednesday, August 19, 2015 1:48 PM
To: user@spark.apache.org
Subject: SparkR csv without headers
Hi,
Does anyone have an example of how to create a DataFrame in SparkR which specifies the column names - the csv files I have do not have column names in the first row. I can get read a csv nicely with com.databricks:spark-csv_2.10:1.0.3, but I end up with column names C1, C2, C3 etc
thanks
--
Franc Carter I Systems Architect I RoZetta Technology
[Description: Description: Description: cid:image003.jpg@01D02903.9B540580]
L4. 55 Harrington Street, THE ROCKS, NSW, 2000
PO Box H58, Australia Square, Sydney NSW, 1215, AUSTRALIA
T +61 2 8355 2515<tel:%2B61%202%208355%202515> I www.rozettatechnology.com<http://www.rozettatechnology.com/>
[cid:image002.jpg@01D02903.0B41B280]
DISCLAIMER: The contents of this email, inclusive of attachments, may be legally
privileged and confidential. Any unauthorised use of the contents is expressly prohibited.