You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Thomas D'Silva (JIRA)" <ji...@apache.org> on 2016/07/22 22:30:20 UTC
[jira] [Comment Edited] (PHOENIX-2995) Write performance severely
degrades with large number of views
[ https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390304#comment-15390304 ]
Thomas D'Silva edited comment on PHOENIX-2995 at 7/22/16 10:29 PM:
-------------------------------------------------------------------
[~jamestaylor], [~samarth.jain@gmail.com]
We were cloning the metadata cache in a connection everytime we wanted to update it. I have attached a patch that only clones the metadata cache if another conneection or query services holds a reference to the cache.
I ran [~mujtaba] 's test and have attached an updated chart. It not takes about 4 seconds to commit 1K rows while we are creating the views and <2 seconds to commit 1K rows after all the views were created.
!image.png!
was (Author: tdsilva):
[~jamestaylor], [~samarth.jain@gmail.com]
We were cloning the metadata cache in a connection everytime we wanted to update it. I have attached a patch that only clones the metadata cache if another conneection or query services holds a reference to the cache.
I ran [~mujtaba] 's test and have attached an updated chart. It not takes about 4 seconds to commit 1K rows while we are creating the views and <2 seconds to commit 1K rows after all the views were created.
> Write performance severely degrades with large number of views
> ---------------------------------------------------------------
>
> Key: PHOENIX-2995
> URL: https://issues.apache.org/jira/browse/PHOENIX-2995
> Project: Phoenix
> Issue Type: Bug
> Reporter: Mujtaba Chohan
> Assignee: Thomas D'Silva
> Labels: Argus
> Fix For: 4.9.0
>
> Attachments: PHOENIX-2995.patch, create_view_and_upsert.png, image.png, upsert_rate.png
>
>
> Write performance for each 1K batch degrades significantly when there are *10K* views being written in random with default {{phoenix.client.maxMetaDataCacheSize}}. With all views created, upsert rate remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate.
> When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view does not need to get re-resolved and upsert rate gets back to normal ~60K rows/min.
> With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I wasn't able create all 100K views as upsert time for each 1K batch keeps on steadily increasing.
> Following graph shows 1K batch upsert rate over time with variation of number of views. Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ... APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000}} is executed before upsert statement.
> !upsert_rate.png!
> Base table is also created with {{APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)