Deploying Cloud Block Store on Google Kubernetes Engine

  1. Create a GKE cluster on Google Cloud Platform (GCP).

  • Go to the “IAM & Admin/Roles” page.
  • Create a role with the role launch stage “General Availability”.
  • Add the following permissions:

 

container.clusterRoleBindings.create

container.clusterRoles.create

container.namespaces.get

container.namespaces.list

container.pods.get

container.pods.list

monitoring.metricDescriptors.create

monitoring.metricDescriptors.get

monitoring.metricDescriptors.list

monitoring.monitoredResourceDescriptors.get

monitoring.monitoredResourceDescriptors.list

monitoring.timeSeries.create

resourcemanager.projects.get

storage.buckets.get

storage.buckets.list

storage.objects.create

storage.objects.delete

storage.objects.get

storage.objects.getIamPolicy

storage.objects.list

storage.objects.setIamPolicy

storage.objects.update

serviceusage.services.use

 

  • Go to the “IAM & Admin/Service Accounts” page.
  • Create a service account.
  • Go to the “IAM & Admin/IAM” page.
  • Associate the role and service accounts you created.
  • Create the cluster with “Master version/Static version 1.17.13-gke.2001”.
  • Create three node pools to represent different components in a real-life cluster.
    1. One node pool will be for DPL, one for BD, and the last for CMAP.
    2. You can select the machine types based on your need, but the following table lists our recommendations for each node pool:

Cassandra/DPL: 

Series: N1

Machine type: n1-highmem-16 (16 vCPU, 104 GB memory)

BD: 

Series: E2

Machine type: e2-highmem-8 (8 vCPU, 64 GB memory)

CMAP: 

Series: N2

Machine type: n2-highmem-8 (8 vCPU, 64 GB memory)

    1. Use Ubuntu as the image type.
    2. The nodes in the pools should be in different zones to ensure high availability.
  • Regarding the node pools:
    1. The DPL node pool’s components “mjcachedpl”, “servicedpl”, and “djcachedpl” require GKE local SSD disks. We recommend attaching a total of 8 local SSD disks.
    2. The CMAP node pool’s components “cmapdcdpl” and “cmapmcdpl” also require GKE local SSD disks. We recommend using “preemptible node” and attaching a total of 4 local SSD disks.
    3. Most image types can attach at most 8 disks. Each disk can be up to 375 GB, reaching 3 TB total. If you need a larger size, create more node pools to split the disks up between, and contact us for more instruction on disk setting during setup.
    4. Creating a private cluster means adding firewall rules for TCP ports 8443 and 6443. Port 8443 is used by the admission webhook, while port 6443 is used by the API server. For details, please refer here
  • Add the following labels to the corresponding node pools:

Cassandra/DPL: cassandra-1=true, mjcachedpl-1-1=true, servicedpl-1-1=true, djcachedpl-17-1-1=true, flushdpl-1=true, jddpl-1=true, jddpl-2=true

BD: bd=true

CMAP: cmapdcdpl-17-1=true, cmapmcdpl-1=true

  • You should be able to see the cluster on the GCP console under Kubernetes clusters.

2. Prepare an extra VM (which should be a bastion host) to connect the GKE cluster to.

  • Install python2.
  • Install the kubectl binary.
  1. You can find it at: https://kubernetes.io/docs/tasks/tools/install-kubectl/
  2. We currently use k8s 1.17.13.
  • Install and initialize the Google Cloud SDK with the following commands:

curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init

  • Grant the VM kubectl command line access.
  1. Locate your GKE cluster on the GCP console.
  2.  Click the Connect button to the right of it.
  3.  Run the displayed command on your VM.
  4. The following is an example of what the command looks like:

gcloud container clusters get-credentials vizion-ai-vbos-cluster --region us-west1 --project evi-vizion-ai

3. Setup VBOS.

  • Save the table below as a .yaml file. It will be used to configure a Kubernetes job that will deploy VBOS.
    • This job will create a new namespace “vizion” and a new service account “vbos”.
    • Replace the italicized “<google cloud storage access key>” and "<google cloud storage secret key>" fields with the appropriate keys for your Google Cloud Storage.
    • Replace the italicized “<your company name>” and "<your company email>" fields with the appropriate substitutes.

---

apiVersion: v1

kind: Namespace

metadata:

  labels:

    name: vbos

  name: vizion

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: vbos

subjects:

- kind: ServiceAccount

  name: vbos

  namespace: vizion

roleRef:

  kind: ClusterRole

  name: cluster-admin

  apiGroup: rbac.authorization.k8s.io

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: vbos

  namespace: vizion

---

apiVersion: v1

kind: Secret

metadata:

  name: imagepullsecret

  namespace: vizion

data:

  .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS52aXppb24uYWkiOnsidXNlcm5hbWUiOiJkZXZvcHMiLCJwYXNzd29yZCI6IjRHOXJzJGU4VTFDQCIsImF1dGgiOiJaR1YyYjNCek9qUkhPWEp6SkdVNFZURkRRQT09In19fQ==

type: kubernetes.io/dockerconfigjson

---

apiVersion: batch/v1

kind: Job

metadata:

  name: vbos-setup

  namespace: vizion

spec:

  backoffLimit: 0

  ttlSecondsAfterFinished: 3600

  template:

    metadata:

      name: vbos-setup

    spec:

      imagePullSecrets:

        - name: imagepullsecret

      serviceAccountName: vbos

      containers:

      - name: vbos-setup

        image: registry.vizion.ai/review/vbos:latest

        command: ['/bin/sh', '-c', 'bash -c "${STARTUP_SCRIPT}"']

        imagePullPolicy: Always

        env:

          - name: SERVER_MODE

            value: "prod"

          - name: COMPANY_NAME

            value: "<your company name>"

          - name: COMPANY_EMAIL

            value: "<your company email>"

          - name: STORAGE_ENDPOINT

            value: "storage.googleapis.com"

          - name: ACCESS_KEY

            value: "<google cloud storage access key>"

          - name: SECRET

            value: "<google cloud storage secret key>"

          - name: STARTUP_SCRIPT

            value: |

              #!/usr/bin/env bash

              set -euo pipefail

              set -x


              if [[ ! -f "/root/setup/cfg/custom.yml" ]]; then

                mkdir -p /root/setup/cfg/

                cli parameter init

              fi


              cli parameter set \

              --key=storage_provider.vset1.connection_type=1 \

              --key=storage_provider.vset1.host=`echo ${STORAGE_ENDPOINT}` \

              --key=storage_provider.vset1.name=evi \

              --key=storage_provider.vset1.password=`echo ${SECRET}` \

              --key=storage_provider.vset1.port=443 \

              --key=storage_provider.vset1.region=104 \

              --key=storage_provider.vset1.user=`echo ${ACCESS_KEY}` \

              --key=storage_provider.vset1.storage_type=2 \

              --key=storage_provider.vset1.vendor_type=2 \

              --key=storage_provider.vset1.bucket_name=vbos-`date "+%Y-%m-%d"` \

              --key=vbos_namespace=vizion \

              --key=company_name=`echo ${COMPANY_NAME}` \

              --key=company_email=`echo ${COMPANY_EMAIL}` \

              --key=cluster_uuid=`cat /proc/sys/kernel/random/uuid`

              err_num=0

              if [ "X${SERVER_MODE}" == "Xdev" ]; then

                cli parameter set --key=cloud_vendor=gce \

                --key=installation_method=online \

                --key=default_registry_fqdn=registry.vizion.ai \

                --key=gce.disks.monitor_volume.size=100 \

                --key=gce.disks.cmap_mc_volume.size=20 \

                --key=gce.disks.multiple_volume.k128.cmap_dc_volume.size=100 \

                --key=gce.disks.mc_volume.size=20 \

                --key=gce.disks.multiple_volume.k128.dc_volume.size=100 \

                --key=gce.disks.service_mc_volumes.0.size=20 \

                --key=gce.disks.service_pxs_volumes.0.size=100 \

                --key=gce.disks.service_storage_cache_volumes.0.size=100 \

                --key=gce.disks.j_volumes.0.size=100 \

                --key=vset.type.vset1=volume,s3 \

                --key=vset.type.vset1groupcmapdc128kdplid=1 \

                --key=vset.type.vset1groupcmapmcdplid=1 \

                --key=vset.type.vset1groupdjcache128kdplid=1 \

                --key=vset.type.vset1groupservicedplid=1 \

                --key=vset.type.vset1groupjddplid=1 \

                --key=vset.type.vset1grouppxsid=1 \

                --key=nfs.backend.ext4.vset1.storageSize=2048Gi \

                --key=nfs.backend.ext4.vset1.storageClassName=csi-ext4-sc-vset1

                err_num=$?

              else

                ansible-playbook -i /root/setup/cfg/hosts.ini -e @/root/setup/cfg/custom.yml -v preset-vbos.yml

                err_num=$?

              fi

              if [ ${err_num} -ne 0 ]; then

                echo "Run vbos preset failed."

                exit 1

              fi


              ansible-playbook -i /root/setup/cfg/hosts.ini -e @/root/setup/cfg/custom.yml -v vizion-vbos.yml

              if [ $? -ne 0 ]; then

                echo "Run vbos setup filed."

                exit 1

              fi


              exit 0


        volumeMounts:

          - name: vbos

            mountPath: /etc/kubernetes/vizion

      volumes:

        - hostPath:

            path: /var/log/vbos

            type: ""

          name: vbos

      restartPolicy: Never                 

      tolerations:

      - key: node-role.kubernetes.io/master

        operator: Exists

        effect: NoSchedule

  • Run the following command on the bastion host to begin setup:

Kubectl apply -f  <vbos job yaml>

4. Confirm that VBOS has been setup properly.

  • All pods should be in either the “Running” or “Completed” states.
  1. You can check by running the command “kubectl get pod” on the bastion host.
  • Storage classes “csi-ext4-sc-vset1” and “nfs-provisioner-vset1” should be present, and usable on your apps.

5. Run the provided nginx-deployment-ext4.yam file to test your VBOS deployment.

  • After startup, you can use VBOS as a web server.
  • Add your index.html file to the root directory of the pod: /usr/share/nginx/html
  • Access the web service using: http://<node_ip>:<node_port>
  • Below is an example yaml file:

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: vbos-nginx-ext4-pvc

  namespace: vizion

spec:

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: 100G

  storageClassName: csi-ext4-sc-vset1  # our default storageclass

  volumeMode: Filesystem

---

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    name: nginx-ext4

  name: nginx-ext4

  namespace: vizion

spec:

  replicas: 1

  selector:

    matchLabels:

      name: nginx-ext4

  strategy:

    rollingUpdate:

      maxSurge: 25%

      maxUnavailable: 25%

    type: RollingUpdate

  template:

    metadata:

      labels:

        name: nginx-ext4

    spec:

      containers:

      - name: nginx

        image: nginx

        imagePullPolicy: IfNotPresent

        tty: true

        volumeMounts:

        - mountPath: "/usr/share/nginx/html"

          name: nginx-storage

        ports:

        - containerPort: 80

          name: "http-server"

      volumes:

      - name: nginx-storage

        persistentVolumeClaim:

          claimName: vbos-nginx-ext4-pvc

      nodeSelector:

        bd: "true"

---

kind: Service

apiVersion: v1

metadata:

  name: nginx-ext4

  namespace: vizion

spec:

  selector:

    name: nginx

  type: ClusterIP

  ports:

  - name: p80

    port: 80

    protocol: TCP

    targetPort: 80