You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by "Arthur.hk.chan@gmail.com" <ar...@gmail.com> on 2014/10/09 03:46:31 UTC
WARN impl.MetricsConfig: Cannot locate configuration
Hi,
I am trying Phoenix Performance test script
There are 4 warnings
1) WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
Question: How to resolve it?
2) WARN util.DynamicClassLoader: Failed to identify the fs of dir hdfs://my_cluster/hbase/lib, ignored
May safely ignore this warnings, am I correct?
3) WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
May safely ignore this warnings, am I correct?
Regards
Arthur
(FYI, full output)
./performance.py z1,z2,z3:/hbase 1000
Phoenix Performance Evaluation Script 1.0
-----------------------------------------
Creating performance table...
14/10/09 09:32:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/09 09:32:19 WARN util.DynamicClassLoader: Failed to identify the fs of dir hdfs://my_cluster/hbase/lib, ignored
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:202)
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:858)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:663)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:415)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:310)
at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:235)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:147)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1510)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1489)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1489)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:129)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.util.PhoenixRuntime.main(PhoenixRuntime.java:138)
14/10/09 09:32:19 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
no rows upserted
Time: 0.167 sec(s)
Query # 1 - Count - SELECT COUNT(1) FROM PERFORMANCE_1000;
Query # 2 - Group By First PK - SELECT HOST FROM PERFORMANCE_1000 GROUP BY HOST;
Query # 3 - Group By Second PK - SELECT DOMAIN FROM PERFORMANCE_1000 GROUP BY DOMAIN;
Query # 4 - Truncate + Group By - SELECT TRUNC(DATE,'DAY') DAY FROM PERFORMANCE_1000 GROUP BY TRUNC(DATE,'DAY');
Query # 5 - Filter + Count - SELECT COUNT(1) FROM PERFORMANCE_1000 WHERE CORE<10;
Generating and upserting data...
.
14/10/09 09:32:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/09 09:32:20 WARN util.DynamicClassLoader: Failed to identify the fs of dir hdfs://my_cluster/hbase/lib, ignored
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:202)
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:858)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:663)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:415)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:310)
at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:235)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:147)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1510)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1489)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1489)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:129)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.util.PhoenixRuntime.main(PhoenixRuntime.java:138)
14/10/09 09:32:20 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
csv columns from database.
CSV Upsert complete. 1000 rows upserted
Time: 0.475 sec(s)
COUNT(1)
----------
1955
Time: 0.031 sec(s)
HOST
----
CS
EU
NA
Time: 0.025 sec(s)
DOMAIN
----------
Apple.com
Google.com
Salesforce.com
Time: 0.053 sec(s)
DAY
-------------------
2014-10-09 00:00:00
Time: 0.054 sec(s)
COUNT(1)
----------
32
Time: 0.027 sec(s)
Re: SALT_BUCKETS
Posted by Mujtaba Chohan <mu...@apache.org>.
In addition to what Gabriel said, it also depends on kind of queries that
you would most frequently execute. If you queries are highly selective, say
the filter just aggregates 1% rows of entire table based on row key then
keeping salt buckets size low or even without salting might give better
performance as there would be fewer blocks read from disk.
For full table aggregation queries and for better write performance higher
value for salt buckets would be better.
Also note with 1tb of data and low value of salt bucket, table would
still get auto-split after region size threshold is reached based on split
policy.
//Mujtaba
On Thursday, October 9, 2014, Gabriel Reid <ga...@gmail.com> wrote:
> Hi Arthur,
>
> Each salt bucket results in a single region by default. You'll
> probably want to have more than one region per region server, although
> this can depend on a number of things. There's some good information
> on choosing the appropriate number of regions here:
> http://hbase.apache.org/book/regions.arch.html
>
> In any case, you'll want at least 5 buckets if you're working with 5
> nodes, and it's a good idea to work in multiples of 5 (i.e. 5, 10, 15,
> or 20) if you want to distribute the data relatively evenly over your
> cluster.
>
> - Gabriel
>
>
> On Thu, Oct 9, 2014 at 9:06 AM, Arthur.hk.chan@gmail.com <javascript:;>
> <arthur.hk.chan@gmail.com <javascript:;>> wrote:
> > Hi,
> >
> > I plan to use Phoenix to load in a very large transaction detail table
> (about 1TB), my HBASE cluster has 5 nodes, I plan to create a SALTED TABLE
> for it.
> >
> > 2 questions:
> > Q1) Can anyone suggest how to determine the proper value for
> SALT_BUCKETS?
> > Q2) In my case here, if I set SALT_BUCKETS = 5 (same number of HBase
> node), is it a proper setting?
> >
> > Regards
> > Arthur
> >
> >
> >
> >
>
Re: SALT_BUCKETS
Posted by Gabriel Reid <ga...@gmail.com>.
Hi Arthur,
Each salt bucket results in a single region by default. You'll
probably want to have more than one region per region server, although
this can depend on a number of things. There's some good information
on choosing the appropriate number of regions here:
http://hbase.apache.org/book/regions.arch.html
In any case, you'll want at least 5 buckets if you're working with 5
nodes, and it's a good idea to work in multiples of 5 (i.e. 5, 10, 15,
or 20) if you want to distribute the data relatively evenly over your
cluster.
- Gabriel
On Thu, Oct 9, 2014 at 9:06 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I plan to use Phoenix to load in a very large transaction detail table (about 1TB), my HBASE cluster has 5 nodes, I plan to create a SALTED TABLE for it.
>
> 2 questions:
> Q1) Can anyone suggest how to determine the proper value for SALT_BUCKETS?
> Q2) In my case here, if I set SALT_BUCKETS = 5 (same number of HBase node), is it a proper setting?
>
> Regards
> Arthur
>
>
>
>
SALT_BUCKETS
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I plan to use Phoenix to load in a very large transaction detail table (about 1TB), my HBASE cluster has 5 nodes, I plan to create a SALTED TABLE for it.
2 questions:
Q1) Can anyone suggest how to determine the proper value for SALT_BUCKETS?
Q2) In my case here, if I set SALT_BUCKETS = 5 (same number of HBase node), is it a proper setting?
Regards
Arthur
Re: How to change default field delimiter from COMMA to SEMICOLON
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
This is one of the option. But I wish to know what the correct -d parameter should be in command line (the testing data set is of TB size)
regards
Arthur
On 9 Oct, 2014, at 1:06 pm, sunfl@certusnet.com.cn wrote:
> Maybe running a program to modify your CSV file to replace any SEMICOLON with COMMA shall be more convinient.
>
>
>
> From: Arthur.hk.chan@gmail.com
> Date: 2014-10-09 11:26
> To: user
> CC: Arthur.hk.chan@gmail.com
> Subject: How to change default field delimiter from COMMA to SEMICOLON
> Hi,
>
> My CSV file uses semicolon as field delimiter, I tried to use -d ; but failed.
>
>
> 1) without -d parameter
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv
> 14/10/09 11:14:21 ERROR util.CSVCommonsLoader: Error upserting record [19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent permanent sauternes-- decoys hang slowly into the sentiments! forges toward"]: java.lang.NumberFormatException: For input string: "19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent permanent sauternes-- decoys hang slowly into the sentiments! forges toward""
>
>
> 2) with -d parameter
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ;
>
> ...
> -d,--delimiter <arg> Field delimiter for CSV loader. A digit is
> interpreted as 1 -> ctrl A, 2 -> ctrl B ...
> 9 -> ctrl I.
> -e,--escape-character <arg> Escape character for CSV loader. A digit is
> interpreted as a control character
>
>
>
> CSV sample data:
> 0;"ARGENTINA";1;"ironic regular realms through the idly thin sauternes could eat boldly regular daring warthogs-- daringly idle somas could have to lo"
> 1;"BRAZIL";1;"silently quiet realms haggle boldly slow ruthless platelets? even i"
> 2;"CANADA";1;"fluffy pinto beans until the asymptotes doze slowly even epitaphs! doggedly busy excuses sublate carefully: quiet brave asymptotes boost sometimes on th"
> 9;"IRAN";4;"warthogs could poach even forges? bold bold attainments among the idly permanent warhorses are permanently in place of the bravely fu"
> 10;"IRAQ";4;"blithe excuses should have to believe; silent busy notornis print toward the slowly furious theodolites. even platelets serve bold ruthless tithes? shea"
> 11;"JAPAN";2;"dolphins can nag! enticingly bold warhorses will unwind never past the grouches; ironic quick s"
>
>
> Q: How to change default field delimiter from COMMA to SEMICOLON in psql.py command line?
>
> Regards
> Arthur
Re: How to change default field delimiter from COMMA to SEMICOLON
Posted by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>.
Maybe running a program to modify your CSV file to replace any SEMICOLON with COMMA shall be more convinient.
From: Arthur.hk.chan@gmail.com
Date: 2014-10-09 11:26
To: user
CC: Arthur.hk.chan@gmail.com
Subject: How to change default field delimiter from COMMA to SEMICOLON
Hi,
My CSV file uses semicolon as field delimiter, I tried to use -d ; but failed.
1) without -d parameter
./psql.py z1:/hbase -t NATION ../sample/NATION.csv
14/10/09 11:14:21 ERROR util.CSVCommonsLoader: Error upserting record [19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent permanent sauternes-- decoys hang slowly into the sentiments! forges toward"]: java.lang.NumberFormatException: For input string: "19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent permanent sauternes-- decoys hang slowly into the sentiments! forges toward""
2) with -d parameter
./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ;
...
-d,--delimiter <arg> Field delimiter for CSV loader. A digit is
interpreted as 1 -> ctrl A, 2 -> ctrl B ...
9 -> ctrl I.
-e,--escape-character <arg> Escape character for CSV loader. A digit is
interpreted as a control character
CSV sample data:
0;"ARGENTINA";1;"ironic regular realms through the idly thin sauternes could eat boldly regular daring warthogs-- daringly idle somas could have to lo"
1;"BRAZIL";1;"silently quiet realms haggle boldly slow ruthless platelets? even i"
2;"CANADA";1;"fluffy pinto beans until the asymptotes doze slowly even epitaphs! doggedly busy excuses sublate carefully: quiet brave asymptotes boost sometimes on th"
9;"IRAN";4;"warthogs could poach even forges? bold bold attainments among the idly permanent warhorses are permanently in place of the bravely fu"
10;"IRAQ";4;"blithe excuses should have to believe; silent busy notornis print toward the slowly furious theodolites. even platelets serve bold ruthless tithes? shea"
11;"JAPAN";2;"dolphins can nag! enticingly bold warhorses will unwind never past the grouches; ironic quick s"
Q: How to change default field delimiter from COMMA to SEMICOLON in psql.py command line?
Regards
Arthur
Re: How to change default field delimiter from COMMA to SEMICOLON
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
Thank you so much!
Arthur
On 9 Oct, 2014, at 3:45 pm, sunfl@certusnet.com.cn wrote:
> Hi, Arthur
> I tested CSV load through psql and found the following delimeter works. Possibly SEMICOLON need an ESCAPE character.
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d "\;"
>
> Sun
>
>
> From: Gabriel Reid
> Date: 2014-10-09 15:17
> To: user
> Subject: Re: How to change default field delimiter from COMMA to SEMICOLON
> Hi,
>
> You've got the usage of the command correct there, but the semi-colon
> character has a special meaning in most shells. Wrapping it with
> single quotes should resolve the issue, as follows:
>
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ';'
>
>
> - Gabriel
>
>
> On Thu, Oct 9, 2014 at 5:26 AM, Arthur.hk.chan@gmail.com
> <ar...@gmail.com> wrote:
> > Hi,
> >
> > My CSV file uses semicolon as field delimiter, I tried to use -d ; but
> > failed.
> >
> >
> > 1) without -d parameter
> > ./psql.py z1:/hbase -t NATION ../sample/NATION.csv
> > 14/10/09 11:14:21 ERROR util.CSVCommonsLoader: Error upserting record
> > [19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle
> > silent permanent sauternes-- decoys hang slowly into the sentiments! forges
> > toward"]: java.lang.NumberFormatException: For input string: "19;"SAUDI
> > ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent
> > permanent sauternes-- decoys hang slowly into the sentiments! forges
> > toward""
> >
> >
> > 2) with -d parameter
> > ./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ;
> >
> > ...
> > -d,--delimiter <arg> Field delimiter for CSV loader. A digit is
> > interpreted as 1 -> ctrl A, 2 -> ctrl B ...
> > 9 -> ctrl I.
> > -e,--escape-character <arg> Escape character for CSV loader. A digit is
> > interpreted as a control character
> >
> >
> >
> > CSV sample data:
> > 0;"ARGENTINA";1;"ironic regular realms through the idly thin sauternes could
> > eat boldly regular daring warthogs-- daringly idle somas could have to lo"
> > 1;"BRAZIL";1;"silently quiet realms haggle boldly slow ruthless platelets?
> > even i"
> > 2;"CANADA";1;"fluffy pinto beans until the asymptotes doze slowly even
> > epitaphs! doggedly busy excuses sublate carefully: quiet brave asymptotes
> > boost sometimes on th"
> > 9;"IRAN";4;"warthogs could poach even forges? bold bold attainments among
> > the idly permanent warhorses are permanently in place of the bravely fu"
> > 10;"IRAQ";4;"blithe excuses should have to believe; silent busy notornis
> > print toward the slowly furious theodolites. even platelets serve bold
> > ruthless tithes? shea"
> > 11;"JAPAN";2;"dolphins can nag! enticingly bold warhorses will unwind never
> > past the grouches; ironic quick s"
> >
> >
> > Q: How to change default field delimiter from COMMA to SEMICOLON in psql.py
> > command line?
> >
> > Regards
> > Arthur
> >
> >
> >
Re: Re: How to change default field delimiter from COMMA to SEMICOLON
Posted by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>.
Hi, Arthur
I tested CSV load through psql and found the following delimeter works. Possibly SEMICOLON need an ESCAPE character.
./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d "\;"
Sun
From: Gabriel Reid
Date: 2014-10-09 15:17
To: user
Subject: Re: How to change default field delimiter from COMMA to SEMICOLON
Hi,
You've got the usage of the command correct there, but the semi-colon
character has a special meaning in most shells. Wrapping it with
single quotes should resolve the issue, as follows:
./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ';'
- Gabriel
On Thu, Oct 9, 2014 at 5:26 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> My CSV file uses semicolon as field delimiter, I tried to use -d ; but
> failed.
>
>
> 1) without -d parameter
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv
> 14/10/09 11:14:21 ERROR util.CSVCommonsLoader: Error upserting record
> [19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle
> silent permanent sauternes-- decoys hang slowly into the sentiments! forges
> toward"]: java.lang.NumberFormatException: For input string: "19;"SAUDI
> ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent
> permanent sauternes-- decoys hang slowly into the sentiments! forges
> toward""
>
>
> 2) with -d parameter
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ;
>
> ...
> -d,--delimiter <arg> Field delimiter for CSV loader. A digit is
> interpreted as 1 -> ctrl A, 2 -> ctrl B ...
> 9 -> ctrl I.
> -e,--escape-character <arg> Escape character for CSV loader. A digit is
> interpreted as a control character
>
>
>
> CSV sample data:
> 0;"ARGENTINA";1;"ironic regular realms through the idly thin sauternes could
> eat boldly regular daring warthogs-- daringly idle somas could have to lo"
> 1;"BRAZIL";1;"silently quiet realms haggle boldly slow ruthless platelets?
> even i"
> 2;"CANADA";1;"fluffy pinto beans until the asymptotes doze slowly even
> epitaphs! doggedly busy excuses sublate carefully: quiet brave asymptotes
> boost sometimes on th"
> 9;"IRAN";4;"warthogs could poach even forges? bold bold attainments among
> the idly permanent warhorses are permanently in place of the bravely fu"
> 10;"IRAQ";4;"blithe excuses should have to believe; silent busy notornis
> print toward the slowly furious theodolites. even platelets serve bold
> ruthless tithes? shea"
> 11;"JAPAN";2;"dolphins can nag! enticingly bold warhorses will unwind never
> past the grouches; ironic quick s"
>
>
> Q: How to change default field delimiter from COMMA to SEMICOLON in psql.py
> command line?
>
> Regards
> Arthur
>
>
>
Re: How to change default field delimiter from COMMA to SEMICOLON
Posted by Gabriel Reid <ga...@gmail.com>.
Hi,
You've got the usage of the command correct there, but the semi-colon
character has a special meaning in most shells. Wrapping it with
single quotes should resolve the issue, as follows:
./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ';'
- Gabriel
On Thu, Oct 9, 2014 at 5:26 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> My CSV file uses semicolon as field delimiter, I tried to use -d ; but
> failed.
>
>
> 1) without -d parameter
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv
> 14/10/09 11:14:21 ERROR util.CSVCommonsLoader: Error upserting record
> [19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle
> silent permanent sauternes-- decoys hang slowly into the sentiments! forges
> toward"]: java.lang.NumberFormatException: For input string: "19;"SAUDI
> ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent
> permanent sauternes-- decoys hang slowly into the sentiments! forges
> toward""
>
>
> 2) with -d parameter
> ./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ;
>
> ...
> -d,--delimiter <arg> Field delimiter for CSV loader. A digit is
> interpreted as 1 -> ctrl A, 2 -> ctrl B ...
> 9 -> ctrl I.
> -e,--escape-character <arg> Escape character for CSV loader. A digit is
> interpreted as a control character
>
>
>
> CSV sample data:
> 0;"ARGENTINA";1;"ironic regular realms through the idly thin sauternes could
> eat boldly regular daring warthogs-- daringly idle somas could have to lo"
> 1;"BRAZIL";1;"silently quiet realms haggle boldly slow ruthless platelets?
> even i"
> 2;"CANADA";1;"fluffy pinto beans until the asymptotes doze slowly even
> epitaphs! doggedly busy excuses sublate carefully: quiet brave asymptotes
> boost sometimes on th"
> 9;"IRAN";4;"warthogs could poach even forges? bold bold attainments among
> the idly permanent warhorses are permanently in place of the bravely fu"
> 10;"IRAQ";4;"blithe excuses should have to believe; silent busy notornis
> print toward the slowly furious theodolites. even platelets serve bold
> ruthless tithes? shea"
> 11;"JAPAN";2;"dolphins can nag! enticingly bold warhorses will unwind never
> past the grouches; ironic quick s"
>
>
> Q: How to change default field delimiter from COMMA to SEMICOLON in psql.py
> command line?
>
> Regards
> Arthur
>
>
>
How to change default field delimiter from COMMA to SEMICOLON
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
My CSV file uses semicolon as field delimiter, I tried to use -d ; but failed.
1) without -d parameter
./psql.py z1:/hbase -t NATION ../sample/NATION.csv
14/10/09 11:14:21 ERROR util.CSVCommonsLoader: Error upserting record [19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent permanent sauternes-- decoys hang slowly into the sentiments! forges toward"]: java.lang.NumberFormatException: For input string: "19;"SAUDI ARABIA";4;"fluffy close warthogs into the fluffy gifts kindle silent permanent sauternes-- decoys hang slowly into the sentiments! forges toward""
2) with -d parameter
./psql.py z1:/hbase -t NATION ../sample/NATION.csv -d ;
...
-d,--delimiter <arg> Field delimiter for CSV loader. A digit is
interpreted as 1 -> ctrl A, 2 -> ctrl B ...
9 -> ctrl I.
-e,--escape-character <arg> Escape character for CSV loader. A digit is
interpreted as a control character
CSV sample data:
0;"ARGENTINA";1;"ironic regular realms through the idly thin sauternes could eat boldly regular daring warthogs-- daringly idle somas could have to lo"
1;"BRAZIL";1;"silently quiet realms haggle boldly slow ruthless platelets? even i"
2;"CANADA";1;"fluffy pinto beans until the asymptotes doze slowly even epitaphs! doggedly busy excuses sublate carefully: quiet brave asymptotes boost sometimes on th"
9;"IRAN";4;"warthogs could poach even forges? bold bold attainments among the idly permanent warhorses are permanently in place of the bravely fu"
10;"IRAQ";4;"blithe excuses should have to believe; silent busy notornis print toward the slowly furious theodolites. even platelets serve bold ruthless tithes? shea"
11;"JAPAN";2;"dolphins can nag! enticingly bold warhorses will unwind never past the grouches; ironic quick s"
Q: How to change default field delimiter from COMMA to SEMICOLON in psql.py command line?
Regards
Arthur