Back to home

Knowledge Base

Cloud Block Store

Deployment

Deploying Cloud Block Store on Google Kubernetes Engine

Create a GKE cluster on Google Cloud Platform (GCP).

Go to the “IAM & Admin/Roles” page.
Create a role with the role launch stage “General Availability”.
Add the following permissions:

container.clusterRoleBindings.create

container.clusterRoles.create

container.namespaces.get

container.namespaces.list

container.pods.get

container.pods.list

monitoring.metricDescriptors.create

monitoring.metricDescriptors.get

monitoring.metricDescriptors.list

monitoring.monitoredResourceDescriptors.get

monitoring.monitoredResourceDescriptors.list

monitoring.timeSeries.create

resourcemanager.projects.get

storage.buckets.get

storage.buckets.list

storage.objects.create

storage.objects.delete

storage.objects.get

storage.objects.getIamPolicy

storage.objects.list

storage.objects.setIamPolicy

storage.objects.update

serviceusage.services.use

Go to the “IAM & Admin/Service Accounts” page.
Create a service account.
Go to the “IAM & Admin/IAM” page.
Associate the role and service accounts you created.
Create the cluster with “Master version/Static version 1.17.13-gke.2001”.
Create three node pools to represent different components in a real-life cluster.

One node pool will be for DPL, one for BD, and the last for CMAP.
You can select the machine types based on your need, but the following table lists our recommendations for each node pool:

Cassandra/DPL:

Series: N1

Machine type: n1-highmem-16 (16 vCPU, 104 GB memory)

BD:

Series: E2

Machine type: e2-highmem-8 (8 vCPU, 64 GB memory)

CMAP:

Series: N2

Machine type: n2-highmem-8 (8 vCPU, 64 GB memory)

Use Ubuntu as the image type.
The nodes in the pools should be in different zones to ensure high availability.

Regarding the node pools:

The DPL node pool’s components “mjcachedpl”, “servicedpl”, and “djcachedpl” require GKE local SSD disks. We recommend attaching a total of 8 local SSD disks.
The CMAP node pool’s components “cmapdcdpl” and “cmapmcdpl” also require GKE local SSD disks. We recommend using “preemptible node” and attaching a total of 4 local SSD disks.
Most image types can attach at most 8 disks. Each disk can be up to 375 GB, reaching 3 TB total. If you need a larger size, create more node pools to split the disks up between, and contact us for more instruction on disk setting during setup.
Creating a private cluster means adding firewall rules for TCP ports 8443 and 6443. Port 8443 is used by the admission webhook, while port 6443 is used by the API server. For details, please refer here.

Add the following labels to the corresponding node pools:

Cassandra/DPL: cassandra-1=true, mjcachedpl-1-1=true, servicedpl-1-1=true, djcachedpl-17-1-1=true, flushdpl-1=true, jddpl-1=true, jddpl-2=true

BD: bd=true

CMAP: cmapdcdpl-17-1=true, cmapmcdpl-1=true

You should be able to see the cluster on the GCP console under Kubernetes clusters.

2. Prepare an extra VM (which should be a bastion host) to connect the GKE cluster to.

Install python2.
Install the kubectl binary.

You can find it at: https://kubernetes.io/docs/tasks/tools/install-kubectl/
We currently use k8s 1.17.13.

Install and initialize the Google Cloud SDK with the following commands:

curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init

Grant the VM kubectl command line access.

Locate your GKE cluster on the GCP console.
Click the Connect button to the right of it.
Run the displayed command on your VM.
The following is an example of what the command looks like:

gcloud container clusters get-credentials vizion-ai-vbos-cluster --region us-west1 --project evi-vizion-ai

3. Setup VBOS.

Save the table below as a .yaml file. It will be used to configure a Kubernetes job that will deploy VBOS.

This job will create a new namespace “vizion” and a new service account “vbos”.
Replace the italicized “<google cloud storage access key>” and "<google cloud storage secret key>" fields with the appropriate keys for your Google Cloud Storage.
Replace the italicized “<your company name>” and "<your company email>" fields with the appropriate substitutes.

---

apiVersion: v1

kind: Namespace

metadata:

labels:

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

subjects:

- kind: ServiceAccount

namespace: vizion

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

---

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: vizion

---

apiVersion: v1

kind: Secret

metadata:

namespace: vizion

data:

.dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS52aXppb24uYWkiOnsidXNlcm5hbWUiOiJkZXZvcHMiLCJwYXNzd29yZCI6IjRHOXJzJGU4VTFDQCIsImF1dGgiOiJaR1YyYjNCek9qUkhPWEp6SkdVNFZURkRRQT09In19fQ==

type: kubernetes.io/dockerconfigjson

---

apiVersion: batch/v1

kind: Job

metadata:

namespace: vizion

spec:

backoffLimit: 0

ttlSecondsAfterFinished: 3600

template:

metadata:

spec:

imagePullSecrets:

- name: imagepullsecret

serviceAccountName: vbos

containers:

- name: vbos-setup

image: registry.vizion.ai/review/vbos:latest

command: ['/bin/sh', '-c', 'bash -c "${STARTUP_SCRIPT}"']

imagePullPolicy: Always

env:

- name: SERVER_MODE

value: "prod"

- name: COMPANY_NAME

value: "<your company name>"

- name: COMPANY_EMAIL

value: "<your company email>"

- name: STORAGE_ENDPOINT

value: "storage.googleapis.com"

- name: ACCESS_KEY

value: "<google cloud storage access key>"

- name: SECRET

value: "<google cloud storage secret key>"

- name: STARTUP_SCRIPT

value: |

#!/usr/bin/env bash

set -euo pipefail

set -x

if [[ ! -f "/root/setup/cfg/custom.yml" ]]; then

mkdir -p /root/setup/cfg/

cli parameter init

cli parameter set \

--key=storage_provider.vset1.connection_type=1 \

--key=storage_provider.vset1.host=`echo ${STORAGE_ENDPOINT}` \

--key=storage_provider.vset1.name=evi \

--key=storage_provider.vset1.password=`echo ${SECRET}` \

--key=storage_provider.vset1.port=443 \

--key=storage_provider.vset1.region=104 \

--key=storage_provider.vset1.user=`echo ${ACCESS_KEY}` \

--key=storage_provider.vset1.storage_type=2 \

--key=storage_provider.vset1.vendor_type=2 \

--key=storage_provider.vset1.bucket_name=vbos-`date "+%Y-%m-%d"` \

--key=vbos_namespace=vizion \

--key=company_name=`echo ${COMPANY_NAME}` \

--key=company_email=`echo ${COMPANY_EMAIL}` \

--key=cluster_uuid=`cat /proc/sys/kernel/random/uuid`

err_num=0

if [ "X${SERVER_MODE}" == "Xdev" ]; then

cli parameter set --key=cloud_vendor=gce \

--key=installation_method=online \

--key=default_registry_fqdn=registry.vizion.ai \

--key=gce.disks.monitor_volume.size=100 \

--key=gce.disks.cmap_mc_volume.size=20 \

--key=gce.disks.multiple_volume.k128.cmap_dc_volume.size=100 \

--key=gce.disks.mc_volume.size=20 \

--key=gce.disks.multiple_volume.k128.dc_volume.size=100 \

--key=gce.disks.service_mc_volumes.0.size=20 \

--key=gce.disks.service_pxs_volumes.0.size=100 \

--key=gce.disks.service_storage_cache_volumes.0.size=100 \

--key=gce.disks.j_volumes.0.size=100 \

--key=vset.type.vset1=volume,s3 \

--key=vset.type.vset1groupcmapdc128kdplid=1 \

--key=vset.type.vset1groupcmapmcdplid=1 \

--key=vset.type.vset1groupdjcache128kdplid=1 \

--key=vset.type.vset1groupservicedplid=1 \

--key=vset.type.vset1groupjddplid=1 \

--key=vset.type.vset1grouppxsid=1 \

--key=nfs.backend.ext4.vset1.storageSize=2048Gi \

--key=nfs.backend.ext4.vset1.storageClassName=csi-ext4-sc-vset1

err_num=$?

else

ansible-playbook -i /root/setup/cfg/hosts.ini -e @/root/setup/cfg/custom.yml -v preset-vbos.yml

err_num=$?

if [ ${err_num} -ne 0 ]; then

echo "Run vbos preset failed."

exit 1

ansible-playbook -i /root/setup/cfg/hosts.ini -e @/root/setup/cfg/custom.yml -v vizion-vbos.yml

if [ $? -ne 0 ]; then

echo "Run vbos setup filed."

exit 1

exit 0

volumeMounts:

- name: vbos

mountPath: /etc/kubernetes/vizion

volumes:

- hostPath:

path: /var/log/vbos

type: ""

restartPolicy: Never

tolerations:

- key: node-role.kubernetes.io/master

operator: Exists

effect: NoSchedule

Run the following command on the bastion host to begin setup:

Kubectl apply -f <vbos job yaml>

4. Confirm that VBOS has been setup properly.

All pods should be in either the “Running” or “Completed” states.

You can check by running the command “kubectl get pod” on the bastion host.

Storage classes “csi-ext4-sc-vset1” and “nfs-provisioner-vset1” should be present, and usable on your apps.

5. Run the provided nginx-deployment-ext4.yam file to test your VBOS deployment.

After startup, you can use VBOS as a web server.
Add your index.html file to the root directory of the pod: /usr/share/nginx/html
Access the web service using: http://<node_ip>:<node_port>
Below is an example yaml file:

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

namespace: vizion

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 100G

storageClassName: csi-ext4-sc-vset1 # our default storageclass

volumeMode: Filesystem

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

namespace: vizion

spec:

replicas: 1

selector:

matchLabels:

strategy:

rollingUpdate:

maxSurge: 25%

maxUnavailable: 25%

type: RollingUpdate

template:

metadata:

labels:

spec:

containers:

- name: nginx

image: nginx

imagePullPolicy: IfNotPresent

tty: true

volumeMounts:

- mountPath: "/usr/share/nginx/html"

ports:

- containerPort: 80

volumes:

- name: nginx-storage

persistentVolumeClaim:

claimName: vbos-nginx-ext4-pvc

nodeSelector:

bd: "true"

---

kind: Service

apiVersion: v1

metadata:

namespace: vizion

spec:

selector:

type: ClusterIP

ports:

- name: p80

port: 80

protocol: TCP

targetPort: 80

Deploying Cloud Block Store on Google Kubernetes Engine

Create a GKE cluster on Google Cloud Platform (GCP).

2. Prepare an extra VM (which should be a bastion host) to connect the GKE cluster to.

3. Setup VBOS.

4. Confirm that VBOS has been setup properly.

5. Run the provided nginx-deployment-ext4.yam file to test your VBOS deployment.