You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/06/16 08:38:39 UTC
[GitHub] [bookkeeper] OlegKashtanov commented on issue #2728: Entry Log GC may get blocked when using entryLogPerLedgerEnabled option
OlegKashtanov commented on issue #2728:
URL: https://github.com/apache/bookkeeper/issues/2728#issuecomment-862169887
I reproduced that issue using Bookkeeper benchmark tool.
The following deployment and test steps:
#### Configuration
- Kubernetes cluster
- 3 ZKs
- 6 BKs. 1 NVME drive per BK (both for Ledger and Journal).
- 1 BK benchmark tool
#### Deployment steps
- Deploy ZK operator:
```
git clone https://github.com/pravega/zookeeper-operator.git
helm install zk zookeeper-operator/charts/zookeeper-operator --set image.tag=0.2.8
```
- Deploy bookkeeper operator
```
kubectl create -f https://raw.githubusercontent.com/pravega/bookkeeper-operator/master/deploy/certificate.yaml
git clone https://github.com/pravega/bookkeeper-operator.git
helm install bk bookkeeper-operator/charts/bookkeeper-operator --set image.tag=0.1.3 --set testmode.enabled=true
```
- Apply the following configmap:
```
kind: ConfigMap
apiVersion: v1
metadata:
name: bk-config-map
data:
# Configuration values can be set as key-value properties
PRAVEGA_CLUSTER_NAME: "pravega"
WAIT_FOR: zookeeper-client:2181
```
- Build Bookie image using docker file https://github.com/pravega/pravega/blob/master/docker/bookkeeper/Dockerfile
- Create Bookie pods using the below manifest. Don't forget to indicate Bookie's image and change Storage classes:
```
apiVersion: "bookkeeper.pravega.io/v1alpha1"
kind: "BookkeeperCluster"
metadata:
name: "bookkeeper"
spec:
version: <BK_VERSION>
zookeeperUri: zookeeper-client:2181
blockOwnerDeletion: false
envVars: bk-config-map
image:
imageSpec:
repository: "<BK_REPOSITORY>"
pullPolicy: IfNotPresent
replicas: 6
resources:
requests:
memory: 4Gi
cpu: 2000m
limits:
memory: 20Gi
cpu: 8000m
storage:
ledgerVolumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "localfile-bk-ledger"
resources:
requests:
storage: 700Gi
journalVolumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "localfile-bk-journal"
resources:
requests:
storage: 700Gi
indexVolumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "ocs-storagecluster-cephfs"
resources:
requests:
storage: 10Gi
autoRecovery: true
options:
useHostNameAsBookieID: "true"
enableStatistics: "true"
statsProviderClass: "org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider"
isForceGCAllowWhenNoSpace: "true"
minorCompactionThreshold: "0.4"
minorCompactionInterval: "1800"
majorCompactionThreshold: "0.8"
majorCompactionInterval: "43200"
numAddWorkerThreads: "8"
entryLogPerLedgerEnabled: "true"
flushInterval: "60000"
gcWaitTime: "600000"
readBufferSizeBytes: "4096"
writeBufferSizeBytes: "524288"
ledgerStorageClass: "org.apache.bookkeeper.bookie.InterleavedLedgerStorage"
journalDirectories: "/bk/journal/j0,/bk/journal/j1,/bk/journal/j2,/bk/journal/j3,/bk/journal/j4,/bk/journal/j5,/bk/journal/j6,/bk/journal/j7"
ledgerDirectories: "/bk/ledgers/"
flushEntrylogBytes: "0"
jvmOptions:
memoryOpts: ["-Xms2g","-Xmx12g","-XX:MaxDirectMemorySize=8g","-XX:+IgnoreUnrecognizedVMOptions","-XX:+UseContainerSupport","-XX:+UnlockDiagnosticVMOptions"]
```
- Build Bookkeeper benchmark image using docker file:
```
FROM ubuntu:18.04
RUN apt-get update
RUN echo deb http://ppa.launchpad.net/openjdk-r/ppa/ubuntu bionic main >| /etc/apt/sources.list.d/openjdk-r-ubuntu-ppa-bionic.list
RUN apt-get -y install openjdk-8-jdk
RUN apt-get -y install git
RUN apt-get -y install maven
RUN apt-get -y install vim
RUN cd /opt; git clone https://github.com/apache/bookkeeper.git --branch release-4.11.1
WORKDIR /opt/bookkeeper
RUN mvn -pl .,bookkeeper-benchmark clean install -Dmaven.test.skip=true
WORKDIR /root
USER root
RUN cd /root
CMD ["/bin/sh", "-c", "while true; do sleep 1; done"]
```
- Deploy Bookkeeper benchmark pod using the below manifest. Don't forget to indicate Image name:
```
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: benchmark-bookkeeper
spec:
# modify replicas according to your case
replicas: 1
selector:
matchLabels:
app: benchmark-bookkeeper
serviceName: "benchmark-bookkeeper"
template:
metadata:
labels:
app: benchmark-bookkeeper
spec:
containers:
- name: benchmark-bookkeeper
image: bk-bmrk:4.11.1
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: "16"
memory: "20Gi"
requests:
cpu: "2"
memory: "4Gi"
env:
- name: BENCHMARK_EXTRA_OPTS
value: -Dorg.apache.bookkeeper.conf.readsystemproperties=true -DzkLedgersRootPath=/pravega/pravega/bookkeeper/ledgers -Xmx7g -XX:MaxDirectMemorySize=8g
#value: -Dorg.apache.bookkeeper.conf.readsystemproperties=true -DzkLedgersRootPath=/pravega/prvg-pravega/bookkeeper/ledgers
- name: BENCHMARK_EXTRA_CLASSPATH
value: /opt/bookkeeper/bookkeeper-benchmark/target/bookkeeper-benchmark-4.11.1.jar
```
- Run Bookkeeper benchmark:
```
kubectl exec -i benchmark-bookkeeper-0 -- /opt/bookkeeper/bookkeeper-benchmark/bin/benchmark writes -zookeeper zookeeper-client:2181 -quorum 3 -ensemble 3 -ackQuorum 3 -entrysize 300000 -ledgers 300 -throttle 10 -skipwarmup true -time 3600
```
- Monitor in any way the Ledger's drive capacity utilization. You will see the permanent increasing of used space. Here the screenshot from Prometheus monitoring (Ledger's drives capacity utilization):
- When Ledger's capacity utilization is 95%, you'll see the following events in Bookkeeper logs:
`2021-06-11 08:10:19,837 - WARN - [BookieWriteThreadPool-OrderedExecutor-7-0:WriteEntryProcessorV3@73] - BookieServer is running as readonly mode, so rejecting the request from the client!`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org