You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by "Hongxu Ma (JIRA)" <ji...@apache.org> on 2017/11/20 06:00:01 UTC
[jira] [Commented] (HAWQ-1552) hawq does not support hdfs storage policy？
[ https://issues.apache.org/jira/browse/HAWQ-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258831#comment-16258831 ]
Hongxu Ma commented on HAWQ-1552:
---------------------------------
Bad news: HAWQ doesn't support it now.
HAWQ implements an own hdfs access lib: libhdfs3, so can't leverage benefits of the latest HDFS feature...
> hawq does not support hdfs storage policy？
> ------------------------------------------
>
> Key: HAWQ-1552
> URL: https://issues.apache.org/jira/browse/HAWQ-1552
> Project: Apache HAWQ
> Issue Type: Bug
> Components: libhdfs
> Reporter: lynn
> Assignee: Radar Lei
>
> 1.The '/ssd" path on HDFS set storage policy ALL_SSD.
> hdfs storagepolicies -setStoragePolicy -path /ssd -policy ALL_SSD
> 2. check
> [hdfs@master1 ~]$ hdfs storagepolicies -getStoragePolicy -path /ssd
> The storage policy of /ssd:
> BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
> 3. put file to hdfs
> hdfs dfs -put dd.txt /ssd/fs_ssd
> 4.check block location
> [hdfs@master1 ~]$ hdfs fsck /ssd/fs_ssd/dd.txt -blocks -locations -files
> decommissioned replica(s) and 0 decommissioning replica(s).
> 0. BP-845848702-192.168.1.130-1496396138316:blk_1075677761_7587369 len=7 repl=3 [DatanodeInfoWithStorage[192.168.1.133:50010,DS-1510d4e4-cfdb-4184-8f47-7417b91f4f5c,{color:red}SSD{color}], DatanodeInfoWithStorage[192.168.1.132:50010,DS-7d498d01-8242-4621-8901-fe397a8196c3,{color:red}SSD{color}], DatanodeInfoWithStorage[192.168.1.134:50010,DS-37c4e804-1b2a-4156-a54c-cecc8393bb09,{color:red}SSD{color}]]
> 5.hawq create filespace fs_ssd and tablespace ts_ssd, fs_ssd point to /ssd/fs_ssd path
> 6.psql create table
> create table p(i int)with(appendonly=true,orientation=parquet,compresstype=snappy) tablespace fs_ssd;
> 7. psql insert data
> insert into p values(1);
> 8. query the file on hdfs
> select c.relname, d.dat2tablespace tablespace_id, d.oid database_id, c.relfilenode table_id
> from pg_database d, pg_class c, pg_namespace n
> where c.relnamespace = n.oid
> and d.datname = current_database()
> and c.relname = 'p';
> relname | tablespace_id | database_id | table_id
> ---------+---------------+-------------+----------
> p | 1021474 | 1021475 | 1037187
> 9. check the file of table "p" locations
> [hdfs@master1 ~]$ hdfs fsck /ssd/fs_ssd/1021474/1021475/1037187/1 -blocks -locations -files
> Connecting to namenode via http://master1.bigdata:50070/fsck?ugi=hdfs&blocks=1&locations=1&files=1&path=%2Fssd%2Ffs_ssd%2F1021474%2F1021475%2F1037187%2F1
> FSCK started by hdfs (auth:SIMPLE) from /192.168.1.130 for path /ssd/fs_ssd/1021474/1021475/1037187/1 at Fri Nov 17 17:26:17 CST 2017
> /ssd/fs_ssd/1021474/1021475/1037187/1 188 bytes, 1 block(s): OK
> 0. BP-845848702-192.168.1.130-1496396138316:blk_1075677763_7587371 len=188 repl=3 [DatanodeInfoWithStorage[192.168.1.134:50010,DS-4be28698-6ebd-4ae0-a515-f3fb5e1293ab,{color:red}DISK{color}], DatanodeInfoWithStorage[192.168.1.133:50010,DS-99d56cac-5af0-483d-b93f-a1bbae038934,{color:red}DISK{color}], DatanodeInfoWithStorage[192.168.1.132:50010,DS-22c09ee4-49ac-47ed-a592-4f0e84776086,{color:red}DISK{color}]]
> The ALL_SSD storage policy doesn't work!!!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)