Apache Pinot Daily Email Digest (2021-10-26)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@diogo.baeder:** Hi folks! Let me ask a question about SQL queries in
Pinot: is it possible to either truncate a table, or at least delete all of
its rows? I'm asking this because I want to be able to do that for
integration-testing my application. Thanks!  
**@g.kishore:** Yes. You can delete all the segments using rest api call  
**@mayanks:** Hi, yes, you can delete all segments from a Pinot table using
rest-api (Swagger).  
**@diogo.baeder:** Ah, got it, thanks! I'll try that :slightly_smiling_face:  
**@diogo.baeder:** It worked just fine, thanks!  
**@mayanks:** Cool, thanks for confirming.  
**@diogo.baeder:** How do I immediately create a new segment after that,
though? If I delete all segments, I'm left with no segment where I can put my
data as part of my tests...  
**@g.kishore:** See base cluster integration test.. @ken had some ideas of
creating a miniPinot cluster for test purpose  
**@diogo.baeder:** Where can I see that? Was that a discussion here in the
channel?  
**@g.kishore:** Yes but we use free slack and it has no history  
**@diogo.baeder:** Oh... that's why I couldn't find it then :disappointed:  
**@diogo.baeder:** Anyway, it's already late here and I'll continue this
tomorrow. Thanks a lot for the help, guys!  
 **@jain.arpit6:** Hi, I am seeing 2 segments in consuming state for my table
and rest are online. Is that expected behavior? Also the consumption is very
slow after a certain number of records 3M. What could be the reason for this?  
**@mayanks:** Number of segments in `CONSUMING` state should be the same as
number of partitions.  
**@jain.arpit6:** Ok  
**@jain.arpit6:** Than its fine... thanks  
**@ssubrama:** If consumption is slow, there can be multiple reasons. Given
enough cpu and memory pinot will consume as fast as the producer produces
data. So, if you see a mismatch, it is either cpu or memory on the consumer
side. Too many partitions in a single host? Too much paging going on? A burst
of queries stole cpu from consuming threads?  
**@mayanks:** I think I also remember seeing a discussion recently where there
was an issue upstream in the kafka broker(s). So you may want to check on that
as well.  
 **@sasha:** @sasha has joined the channel  
 **@sandeep.hadoopadmn:** Hi Team, We are planning to build Pinot cluster, and
the data from Kafka will be 1000 TPS, So what would be the best cluster spec
will look like? like how many nodes we required, memory, cores?  
**@mayanks:** Would need a bit more info on this such as read qps, data size,
etc.  
**@sandeep.hadoopadmn:** Ack  
 **@yawei.li:** @yawei.li has joined the channel  
 **@abhishek:** Hi @all can any one please add me to the Thirdeye slack
workspace, I am trying to do the setup and the docs look bit old. Facing
issues with installation  
**@g.kishore:** @pyne.suvodeep ^^  
**@pyne.suvodeep:** Hi @abhishek Did you find this:  If the master is giving
trouble, can you try this fork  and let me know  
 **@ruicong.xie:** @ruicong.xie has joined the channel  

###  _#random_

  
 **@sasha:** @sasha has joined the channel  
 **@yawei.li:** @yawei.li has joined the channel  
 **@ruicong.xie:** @ruicong.xie has joined the channel  

###  _#feat-upsert_

  
 **@kavin.kuppusamy:** @kavin.kuppusamy has joined the channel  

###  _#troubleshooting_

  
 **@girishpatel.techbiz:** Hi Team, I have a SaaS service which will keep data
globally with data owners across regions. I want to keep their data in
respective regions to comply with Data Privacy Laws & Regulations. But I have
to run queries globally to do analytics across regions. According to me, there
are two ways to do it. 1\. Have multi-region Pinot setup with real-
time/offline servers localized to specific region. And brokers from across
regions will call servers to run queries. 2\. Have region wise multiple Pinot
setups. Do querying and aggregating result at application level. Any other
approach possible ? Which approach looks more feasible and easier to implement
? Thanks !  
**@richard892:** I think you will end up paying for a lot of egress if you do
scatter gather queries across regions, so I would lean towards option 2
because it will probably lead to smaller amounts of data crossing regions  
**@g.kishore:** Will you have any requirements to aggregate across the two
regions?  
**@g.kishore:** Or will the query have a filter on customerid?  
**@girishpatel.techbiz:** We have requirement of aggregation also across
regions.  
**@g.kishore:** how does privacy work in that case?  
**@girishpatel.techbiz:** Various compliance forbid to store data in their
geographic boundary but not while reading data.  
**@girishpatel.techbiz:** We can do read time aggregation and serve in any
other region.  
**@g.kishore:** got it  
**@g.kishore:** I dont know about your setup, but if cost is not a big factor
and if your queries are mostly aggregation then 1 is better  
**@g.kishore:** 2 is a slippery slope and you might have to rewrite the broker
code in Pinot  
**@girishpatel.techbiz:** @richard892, option 2 will require designing and
development at application level. We will not be able to leverage broker for
aggregation. Can prestodb be another option ?  
**@girishpatel.techbiz:** @g.kishore agreed.  
**@girishpatel.techbiz:** For option 1, is there any mechanism to do segment
assignment based on partition ?  
**@g.kishore:** yes, you can control the assignment based on partition  
**@girishpatel.techbiz:** Any documentation referring how to do it ?  
**@g.kishore:** prestodb is a good workaround for option 2 but that will still
incur the cost assuming prestodb spans across the regions  
**@g.kishore:** so, if you are ok with prestodb, then you should be
theoretically ok with option 1 as well  
**@g.kishore:** @jackie.jxt ^^ where are the docs for partition based
assigment  
**@girishpatel.techbiz:** For option 1, costing is due to network only or any
other factor for costing ?  
**@jackie.jxt:**  
**@girishpatel.techbiz:** My requirement is to keep a partition and it's
replica limited to specific region's servers. I couldn't find how to do it in
above documentation.  
**@girishpatel.techbiz:** I went through instance assignment also  
**@jackie.jxt:** I believe the requirement is to do partition assignment
across multiple clusters. That has to be controlled on the client side by
pushing different segments to different cluster  
**@jackie.jxt:** Segment assignment can only manage the assignment within a
single cluster. I don't know if it is possible to maintain a multi-region
cluster, and all of them sharing the same controller  
**@g.kishore:** @jackie.jxt this is for option 1 - Have multi-region Pinot
setup with real-time/offline servers localized to specific region. And brokers
from across regions will call servers to run queries.  
**@jackie.jxt:** Understood, but the segment assignment can only be applied to
single cluster. It cannot assign segments across multiple clusters  
**@girishpatel.techbiz:** Can we setup single multi-region cluster ?  
**@mayanks:** @jackie.jxt I think there's just one Pinot cluster, but servers
are located in different regions. Also, the requirement (if I am reading
correctly) is that servers on a specific region host specific segments  
**@jackie.jxt:** The potential problem of having one cross region cluster is
the latency between controllers and instances in other region, also the ZK  
**@jackie.jxt:** If the multi-region cluster setup works, then segment
assignment can assign segments based on the partition id to different regions  
**@mayanks:** Yeah, there will be some latency and egress cost. but barring
that, if we use partition based segment assignment such that data for a region
goes to servers for that region, isn't that what we need here?  
**@girishpatel.techbiz:** Yes, that is the requirement.  
**@girishpatel.techbiz:** @jackie.jxt how can we assign segment based on
partition id ? Is there any documentation for that ?  
**@jackie.jxt:** The segment and instance assignment doc you just posted  
**@jackie.jxt:** You might need to manually set the `InstancePartitions`
(mapping from (partition id & replica id) to servers)). Pinot won't know which
server is in which region automatically  
**@girishpatel.techbiz:** Need little direction here, how to set
`InstancePartitions` ? Can I set any configuration or need to do something
else ?  
**@mayanks:** @jackie.jxt ^^  
**@jackie.jxt:** Here is the rest api signature: ``` @PUT
@Produces(MediaType.APPLICATION_JSON)
@Path("/tables/{tableName}/instancePartitions")
@Authenticate(AccessType.UPDATE) @ApiOperation(value = "Create/update the
instance partitions") public Map<InstancePartitionsType, InstancePartitions>
setInstancePartitions( @ApiParam(value = "Name of the table")
@PathParam("tableName") String tableName, String instancePartitionsStr)``` You
may check the `InstancePartitions.java` for details  
 **@tamas.nadudvari:** Hello, we’re trying to use pool based assignment. We
have three availability zones, a server group in each with different pool
tags. The table replication setting is 2. Our issue is that we either have a
three sized replica group and the segments will be replicated to each pool
(unnecessary increasing the table replication) or a 2 sized replica group but
then one of the server groups won’t have any segments to serve. Any idea for a
configuration where each server groups are used and the replication of the
table remains 2?  
**@tamas.nadudvari:** this is the configuration when we end up increased
replication: ```"instanceAssignmentConfigMap": { "OFFLINE": { "tagPoolConfig":
{ "tag": "DefaultTenant_OFFLINE", "poolBased": true, "numPools": 0 },
"replicaGroupPartitionConfig": { "replicaGroupBased": true, "numInstances": 0,
"numReplicaGroups": 3, "numInstancesPerReplicaGroup": 0, "numPartitions": 1,
"numInstancesPerPartition": 2 } } },```  
**@g.kishore:** Would you mind describing the problem you are trying to solve  
**@g.kishore:** Pool based is a bit advanced feature  
**@tamas.nadudvari:** We’re running Pinot in Kubernetes, and we’d like to
achieve zero downtime upgrade. Thus we created independent pinot server
Kubernetes deployments for each availability zone, all of them are members of
the same Pinot cluster. We’d like to create a table configuration where a
segment and its replica are in a different availability zone(/Kubernetes
deployment) in any case. We’re trying to use pool based assignments for this.
Servers in the same zone are in the same pool. But so far we were only able to
create configs where either a segments are available in every zone which
increases our desired replica size for the table or using only two of the
server groups leaving one server group completely unused.  
 **@diogo.baeder:** Hi folks, I'd like to ask for a bit of help here: I'm
still struggling to be able to run integration tests between my app and Pinot.
What I need is to be able to start each test with a clean slate, while also
making sure I have the tables I need created. I tried deleting the tables and
recreating them, tried deleting all segments then creating the tables
afterwards (hoping a new segment would automatically be created), and none of
these methods are working. How can I accomplish what I need?  
**@diogo.baeder:** Also another problem I'm facing is, if I try to delete the
table and then recreate it, and then add data to it, it seems like the table
was never deleted, because in the "Incubator" (the Pinot admin UI) I can see
all the previously created rows.  
**@g.kishore:** Delete happens asynchronously.. you will have to wait in your
code for query to return zero records  
**@diogo.baeder:** So, how would you put all of this together, in steps? Would
this make sense?: 1\. Delete all tables 2\. Wait a few secs 3\. Recreate all
tables 4\. Wait a few secs 5\. Start running the test (publishing rows and
querying them afterwards)  
**@g.kishore:** I would not make it wait based on time  
**@g.kishore:** Instead wait for count * queries to return 0  
**@g.kishore:** See baseclusterintegrationtest in Pinot code  
**@g.kishore:** It has everything you need  
**@diogo.baeder:** Nice, I'll look for that. Thanks man!  
**@mayanks:** Also, if your tests are table specific, then you can create
different tables for different tests in the same cluster. That way one test
doesn't have to wait for the previous test to complete and delete the table.  
**@diogo.baeder:** Got it. It's going to be 2 tables total, actually, we're
going to use it for 2 parts of our internal analytics, so I'll work on having
them cleaned up before each test. Thanks!  
**@diogo.baeder:** Alright, so, with this:  If I understand correctly it
starts all the applications and then tears them down around each test, also
deleting the temp dir used to store the data afterwards. However this is not
very efficient for me, I'd like to avoid having to start and stop the whole
suite of apps on each test, if possible.  
**@g.kishore:** you can use beforeSuite for setup then  
**@g.kishore:** in each setup for testcase you can delete the segments if
table exists and wait for the count to go to zero  
**@diogo.baeder:** Hmmm... not sure I understand that last part ("delete the
table if it exists and wait for it go to zero"), but how about I: 1\. Delete
the table 2\. Keep listing tables until my table doesn't exist 3\. Create the
table 4\. Keep counting until I get result zero  
**@g.kishore:** edited - in each setup for testcase you can delete the
segments if table exists and wait for the count to go to zero (edited)  
**@g.kishore:** I think your flow may not work because the segments might
still be on the servers  
**@g.kishore:** delete segments is asynchronous  
**@diogo.baeder:** Hmmm... got it... but after I delete all the segments, how
do new ones get created? Last time I deleted all the segments in the table I
got no new ones created...  
 **@sasha:** @sasha has joined the channel  
 **@yawei.li:** @yawei.li has joined the channel  
 **@ruicong.xie:** @ruicong.xie has joined the channel  

###  _#pinot-dev_

  
 **@girishpatel.techbiz:** @girishpatel.techbiz has joined the channel  

###  _#thirdeye-pinot_

  
 **@kavin.kuppusamy:** @kavin.kuppusamy has joined the channel  

###  _#getting-started_

  
 **@girishpatel.techbiz:** @girishpatel.techbiz has joined the channel  

###  _#pinot-rack-awareness_

  
 **@lars-kristian_svenoy:** @lars-kristian_svenoy has joined the channel  
 **@lars-kristian_svenoy:** :eyes: Found this channel in a random mailing list
when trying to figure out how to enable rack awareness for pinot + kafka.. Is
this feature available?  
 **@g.kishore:** Hi Lars, you can ask this in troubleshooting  

### _#fix-numerical-predicate_

  
 **@kavin.kuppusamy:** @kavin.kuppusamy has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org