Apache Pinot Daily Email Digest (2022-04-17)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#troubleshooting_

  
 **@diogo.baeder:** Hi guys, I'm having an issue running Pinot 0.10 locally
where, sometimes after I stop my local docker-compose services and start them
again, my segments go missing and marked as in "bad" status. It doesn't happen
all the time, like now I tested and after the 3rd restart of the services I
managed to have them in a "bad" state. I compared directories before and after
and couldn't find out what's happening that caused them to "spoil". Any clues
why this might be happening? More details in the thread here.  
**@diogo.baeder:** Log file for the docker-compose execution:  
**@diogo.baeder:** My docker-compose file:  
**@diogo.baeder:** The segments are being saved to `/tmp/data/segments`, which
is mounted in the containers so that I don't lose the data between restarts.  
**@mayanks:** Are you tearing down the entire cluster, including ZK? If so you
will lose data  
**@diogo.baeder:** Well, it's just a local experiment really, not a production
cluster. I'm aware that in production I shouldn't have it running like that,
of course.  
**@diogo.baeder:** For development, however, it's annoying to keep losing my
test data, I'd like to make sure it will always be there after I restart the
containers.  
**@mayanks:** If you’d like durablity then do a rolling restart or ensure ZK
and deep store are not completely shutdown  
**@diogo.baeder:** What's the order of when the components can go down? I
cannot afford keeping ZK running forever (since it's a test cluster running in
my personal computer), but "deep store" is easy to ensure since I just mount a
directory as a volume there to keep it between restarts. Currently I have: •
ZK starting first • Controller depending on ZK (so starting second) • Broker
and Server depending on Controller (so starting third)  
**@diogo.baeder:** This is the order in which docker-compose is stopping them
now: ```^CGracefully stopping... (press Ctrl+C again to force) [+] Running 5/5
⠿ Container pinot-broker-xavier Stopped 10.4s ⠿ Container trino-server-xavier
Stopped 0.4s ⠿ Container pinot-server-xavier Stopped 0.6s ⠿ Container pinot-
controller-xavier Stopped 1.0s ⠿ Container zookeeper-xavier Stopped 0.6s```
does it make sense?  
**@mayanks:** Shutdown order server broker controller zk  
**@mayanks:** You can snapshot ZK and bring it back up from snapshot  
**@diogo.baeder:** Currently I'm mounting a volume for ZK, instead of having
to manage snapshots or anything like that. Locally it's easier to handle.  
**@diogo.baeder:** Thanks for the component order, I'll change that here!  
**@mayanks:** If zk state isn’t snapshoted and everything is torn apart, it
will lead to data loss. Not just for Pinot, any system out there.  
**@diogo.baeder:** I was taking a look at the ZK image docs, and it seems that
the snapshots are saved to `/data`, and I have both `/data` and `/datalog`
mounted as volumes, so I believe I should be safe.  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org