You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Peter Skomoroch (JIRA)" <ji...@apache.org> on 2009/07/14 18:53:14 UTC
[jira] Created: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
ctrl-A is the only output delimiter used, regardless of the Hive table structure
--------------------------------------------------------------------------------
Key: HIVE-634
URL: https://issues.apache.org/jira/browse/HIVE-634
Project: Hadoop Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.3.0
Reporter: Peter Skomoroch
Priority: Minor
No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
where daily_timelines is defined as tab delimited
CREATE TABLE daily_timelines (
page_id BIGINT,
dates STRING,
pageviews STRING,
total_pageviews BIGINT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
This page also indicates Hive uses a fixed delimiter, and should be updated:
http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734392#action_12734392 ]
Zheng Shao commented on HIVE-634:
---------------------------------
Good observation. The LazySimpleSerDe that is used to serialize the data already support customized "serialization.null.format" - we just need to expose that functionality to our user.
> ctrl-A is the only output delimiter used, regardless of the Hive table structure
> --------------------------------------------------------------------------------
>
> Key: HIVE-634
> URL: https://issues.apache.org/jira/browse/HIVE-634
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.3.0
> Reporter: Peter Skomoroch
> Priority: Minor
>
> No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
> INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
> where daily_timelines is defined as tab delimited
> CREATE TABLE daily_timelines (
> page_id BIGINT,
> dates STRING,
> pageviews STRING,
> total_pageviews BIGINT)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> This page also indicates Hive uses a fixed delimiter, and should be updated:
> http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731565#action_12731565 ]
Ashish Thusoo commented on HIVE-634:
------------------------------------
+1 to the proposal.
> ctrl-A is the only output delimiter used, regardless of the Hive table structure
> --------------------------------------------------------------------------------
>
> Key: HIVE-634
> URL: https://issues.apache.org/jira/browse/HIVE-634
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.3.0
> Reporter: Peter Skomoroch
> Priority: Minor
>
> No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
> INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
> where daily_timelines is defined as tab delimited
> CREATE TABLE daily_timelines (
> page_id BIGINT,
> dates STRING,
> pageviews STRING,
> total_pageviews BIGINT)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> This page also indicates Hive uses a fixed delimiter, and should be updated:
> http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734008#action_12734008 ]
Zheng Shao commented on HIVE-634:
---------------------------------
There won't be any difference - it's kind of a syntax sugar.
> ctrl-A is the only output delimiter used, regardless of the Hive table structure
> --------------------------------------------------------------------------------
>
> Key: HIVE-634
> URL: https://issues.apache.org/jira/browse/HIVE-634
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.3.0
> Reporter: Peter Skomoroch
> Priority: Minor
>
> No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
> INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
> where daily_timelines is defined as tab delimited
> CREATE TABLE daily_timelines (
> page_id BIGINT,
> dates STRING,
> pageviews STRING,
> total_pageviews BIGINT)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> This page also indicates Hive uses a fixed delimiter, and should be updated:
> http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731319#action_12731319 ]
Zheng Shao commented on HIVE-634:
---------------------------------
The format of the local file has nothing to do with the format of the source table.
We should add the formatting sub clause for insert overwrite.
{code}
INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines'
[ ROW FORMAT DELIMITED | SERDE ... ]
[ FILE FORMAT ...]
SELECT * FROM daily_timelines;
{code}
> ctrl-A is the only output delimiter used, regardless of the Hive table structure
> --------------------------------------------------------------------------------
>
> Key: HIVE-634
> URL: https://issues.apache.org/jira/browse/HIVE-634
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.3.0
> Reporter: Peter Skomoroch
> Priority: Minor
>
> No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
> INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
> where daily_timelines is defined as tab delimited
> CREATE TABLE daily_timelines (
> page_id BIGINT,
> dates STRING,
> pageviews STRING,
> total_pageviews BIGINT)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> This page also indicates Hive uses a fixed delimiter, and should be updated:
> http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
Posted by "Neal Richter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733995#action_12733995 ]
Neal Richter commented on HIVE-634:
-----------------------------------
What would be the output difference between the proposed feature and doing a two step process as follows?
DROP TABLE tmp_exports_daily_timelines;
CREATE TABLE tmp_exports_daily_timelines (
page_id BIGINT,
dates STRING,
pageviews STRING,
total_pageviews BIGINT
) ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION '/tmp/exports/daily_timelines';
INSERT OVERWRITE TABLE tmp_exports_daily_timelines
SELECT page_id, dates, pageviews, total_pageviews
FROM daily_timelines
[ some WHERE clause here]
> ctrl-A is the only output delimiter used, regardless of the Hive table structure
> --------------------------------------------------------------------------------
>
> Key: HIVE-634
> URL: https://issues.apache.org/jira/browse/HIVE-634
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.3.0
> Reporter: Peter Skomoroch
> Priority: Minor
>
> No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
> INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
> where daily_timelines is defined as tab delimited
> CREATE TABLE daily_timelines (
> page_id BIGINT,
> dates STRING,
> pageviews STRING,
> total_pageviews BIGINT)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> This page also indicates Hive uses a fixed delimiter, and should be updated:
> http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-634) ctrl-A is the only output delimiter
used, regardless of the Hive table structure
Posted by "Neal Richter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734147#action_12734147 ]
Neal Richter commented on HIVE-634:
-----------------------------------
Note that both methods seem to use '\N' in the output files instead of 'NULL'.
It would be better for the export to use 'NULL' for full compatibility with MySQL and postgres. I tested '\N' and it's not recognized as 'NULL' with using MySQL's LOAD DATA INFILE. Easy enough for sed to fix with another pass through the data... at the cost of another pass through the data.
Can one also accomplish this goal with a custom Transform?
http://wiki.apache.org/hadoop/Hive/LanguageManual/Transform
> ctrl-A is the only output delimiter used, regardless of the Hive table structure
> --------------------------------------------------------------------------------
>
> Key: HIVE-634
> URL: https://issues.apache.org/jira/browse/HIVE-634
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.3.0
> Reporter: Peter Skomoroch
> Priority: Minor
>
> No matter what the table format, INSERT OVERWRITE LOCAL DIRECTORY will always use ctrl-A delimiters ('\001' ).
> INSERT OVERWRITE LOCAL DIRECTORY '/mnt/daily_timelines' SELECT * FROM daily_timelines;
> where daily_timelines is defined as tab delimited
> CREATE TABLE daily_timelines (
> page_id BIGINT,
> dates STRING,
> pageviews STRING,
> total_pageviews BIGINT)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> This page also indicates Hive uses a fixed delimiter, and should be updated:
> http://wiki.apache.org/hadoop/Hive/LanguageManual/DML
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.