You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bahir.apache.org by "ShiminHuang (Jira)" <ji...@apache.org> on 2020/12/25 09:26:00 UTC
[jira] [Resolved] (BAHIR-255) Kudu flink-sql add hash partitions
nums config&onwer config
[ https://issues.apache.org/jira/browse/BAHIR-255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ShiminHuang resolved BAHIR-255.
-------------------------------
Resolution: Fixed
> Kudu flink-sql add hash partitions nums config&onwer config
> -----------------------------------------------------------
>
> Key: BAHIR-255
> URL: https://issues.apache.org/jira/browse/BAHIR-255
> Project: Bahir
> Issue Type: Improvement
> Components: Flink Streaming Connectors
> Affects Versions: Flink-Next
> Environment: maxos
> Reporter: ShiminHuang
> Priority: Major
> Labels: flink-kudu, hash-partition-nums
> Fix For: Flink-Next
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> h1. Hash Partition Feature
> * The default number of buckets in the previous hash partition was 2* replicas, and there were some problems in the production environment, so I felt I should provide the displayed hash partition configuration.
> * Add the hash partition number configuration 'kudu.hash-partition-nums',which defaults to 3 if not specified.
> h1. Singleton Test
>
> {code:java}
> public class NewTablePropertiesTest extends KuduCatalogTest{
> private KuduCatalog catalog;
> private StreamTableEnvironment tableEnv;
> @BeforeEach
> public void init() {
> StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
> catalog = new KuduCatalog(harness.getMasterAddressesAsString());
> tableEnv = KuduTableTestUtils.createTableEnvWithBlinkPlannerStreamingMode(env);
> tableEnv.registerCatalog("kudu", catalog);
> tableEnv.useCatalog("kudu");
> }
> @Test
> public void testHashPartitionNums() throws TableNotExistException {
> catalog.dropTable(ObjectPath.fromString(EnvironmentSettings.DEFAULT_BUILTIN_DATABASE + ".TestTable1"), true);
> catalog.dropTable(ObjectPath.fromString(EnvironmentSettings.DEFAULT_BUILTIN_DATABASE + ".TestTable2"), true);
> tableEnv.executeSql("CREATE TABLE TestTable1 (`first` STRING, `second` String) WITH ('kudu.hash-columns' = 'first', 'kudu.primary-key-columns' = 'first')");
> tableEnv.executeSql("CREATE TABLE TestTable2 (`first` STRING, `second` String) WITH ('kudu.hash-columns' = 'first','kudu.hash-partition-nums'='6', 'kudu.primary-key-columns' = 'first')");
> }
> @Test
> public void testOwner() throws TableNotExistException {
> catalog.dropTable(ObjectPath.fromString(EnvironmentSettings.DEFAULT_BUILTIN_DATABASE + ".TestTable1"), true);
> tableEnv.executeSql("CREATE TABLE TestTable1 (`first` STRING, `second` String) WITH ('kudu.hash-columns' = 'first', 'kudu.primary-key-columns' = 'first','kudu.table-owner'='admin')");
> }
> }
> {code}
> h1. Other Ideas
> * Regarding the range partition, I think a JSON List can be passed in to describe the corresponding range partition, and I think the hash partition and range partition must exist in one
> h1.
> h1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)