-
Create a GKE cluster on Google Cloud Platform (GCP).
- Go to the “IAM & Admin/Roles” page.
- Create a role with the role launch stage “General Availability”.
- Add the following permissions:
|
container.clusterRoleBindings.create container.clusterRoles.create container.namespaces.get container.namespaces.list container.pods.get container.pods.list monitoring.metricDescriptors.create monitoring.metricDescriptors.get monitoring.metricDescriptors.list monitoring.monitoredResourceDescriptors.get monitoring.monitoredResourceDescriptors.list monitoring.timeSeries.create resourcemanager.projects.get storage.buckets.get storage.buckets.list storage.objects.create storage.objects.delete storage.objects.get storage.objects.getIamPolicy storage.objects.list storage.objects.setIamPolicy storage.objects.update serviceusage.services.use |
- Go to the “IAM & Admin/Service Accounts” page.
- Create a service account.
- Go to the “IAM & Admin/IAM” page.
- Associate the role and service accounts you created.
- Create the cluster with “Master version/Static version 1.17.13-gke.2001”.
- Create three node pools to represent different components in a real-life cluster.
- One node pool will be for DPL, one for BD, and the last for CMAP.
- You can select the machine types based on your need, but the following table lists our recommendations for each node pool:
|
Cassandra/DPL: Series: N1 Machine type: n1-highmem-16 (16 vCPU, 104 GB memory) |
|
BD: Series: E2 Machine type: e2-highmem-8 (8 vCPU, 64 GB memory) |
|
CMAP: Series: N2 Machine type: n2-highmem-8 (8 vCPU, 64 GB memory) |
- Use Ubuntu as the image type.
- The nodes in the pools should be in different zones to ensure high availability.
- Regarding the node pools:
- The DPL node pool’s components “mjcachedpl”, “servicedpl”, and “djcachedpl” require GKE local SSD disks. We recommend attaching a total of 8 local SSD disks.
- The CMAP node pool’s components “cmapdcdpl” and “cmapmcdpl” also require GKE local SSD disks. We recommend using “preemptible node” and attaching a total of 4 local SSD disks.
- Most image types can attach at most 8 disks. Each disk can be up to 375 GB, reaching 3 TB total. If you need a larger size, create more node pools to split the disks up between, and contact us for more instruction on disk setting during setup.
- Creating a private cluster means adding firewall rules for TCP ports 8443 and 6443. Port 8443 is used by the admission webhook, while port 6443 is used by the API server. For details, please refer here.
- Add the following labels to the corresponding node pools:
|
Cassandra/DPL: cassandra-1=true, mjcachedpl-1-1=true, servicedpl-1-1=true, djcachedpl-17-1-1=true, flushdpl-1=true, jddpl-1=true, jddpl-2=true |
|
BD: bd=true |
|
CMAP: cmapdcdpl-17-1=true, cmapmcdpl-1=true |
- You should be able to see the cluster on the GCP console under Kubernetes clusters.
2. Prepare an extra VM (which should be a bastion host) to connect the GKE cluster to.
- Install python2.
- Install the kubectl binary.
- You can find it at: https://kubernetes.io/docs/tasks/tools/install-kubectl/
- We currently use k8s 1.17.13.
- Install and initialize the Google Cloud SDK with the following commands:
|
curl https://sdk.cloud.google.com | bash |
- Grant the VM kubectl command line access.
- Locate your GKE cluster on the GCP console.
- Click the Connect button to the right of it.
- Run the displayed command on your VM.
- The following is an example of what the command looks like:
|
gcloud container clusters get-credentials vizion-ai-vbos-cluster --region us-west1 --project evi-vizion-ai |
3. Setup VBOS.
- Save the table below as a .yaml file. It will be used to configure a Kubernetes job that will deploy VBOS.
- This job will create a new namespace “vizion” and a new service account “vbos”.
- Replace the italicized “<google cloud storage access key>” and "<google cloud storage secret key>" fields with the appropriate keys for your Google Cloud Storage.
- Replace the italicized “<your company name>” and "<your company email>" fields with the appropriate substitutes.
|
--- apiVersion: v1 kind: Namespace metadata: labels: name: vbos name: vizion --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: vbos subjects: - kind: ServiceAccount name: vbos namespace: vizion roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: ServiceAccount metadata: name: vbos namespace: vizion --- apiVersion: v1 kind: Secret metadata: name: imagepullsecret namespace: vizion data: .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS52aXppb24uYWkiOnsidXNlcm5hbWUiOiJkZXZvcHMiLCJwYXNzd29yZCI6IjRHOXJzJGU4VTFDQCIsImF1dGgiOiJaR1YyYjNCek9qUkhPWEp6SkdVNFZURkRRQT09In19fQ== type: kubernetes.io/dockerconfigjson --- apiVersion: batch/v1 kind: Job metadata: name: vbos-setup namespace: vizion spec: backoffLimit: 0 ttlSecondsAfterFinished: 3600 template: metadata: name: vbos-setup spec: imagePullSecrets: - name: imagepullsecret serviceAccountName: vbos containers: - name: vbos-setup image: registry.vizion.ai/review/vbos:latest command: ['/bin/sh', '-c', 'bash -c "${STARTUP_SCRIPT}"'] imagePullPolicy: Always env: - name: SERVER_MODE value: "prod" - name: COMPANY_NAME value: "<your company name>" - name: COMPANY_EMAIL value: "<your company email>" - name: STORAGE_ENDPOINT value: "storage.googleapis.com" - name: ACCESS_KEY value: "<google cloud storage access key>" - name: SECRET value: "<google cloud storage secret key>" - name: STARTUP_SCRIPT value: | #!/usr/bin/env bash set -euo pipefail set -x if [[ ! -f "/root/setup/cfg/custom.yml" ]]; then mkdir -p /root/setup/cfg/ cli parameter init fi cli parameter set \ --key=storage_provider.vset1.connection_type=1 \ --key=storage_provider.vset1.host=`echo ${STORAGE_ENDPOINT}` \ --key=storage_provider.vset1.name=evi \ --key=storage_provider.vset1.password=`echo ${SECRET}` \ --key=storage_provider.vset1.port=443 \ --key=storage_provider.vset1.region=104 \ --key=storage_provider.vset1.user=`echo ${ACCESS_KEY}` \ --key=storage_provider.vset1.storage_type=2 \ --key=storage_provider.vset1.vendor_type=2 \ --key=storage_provider.vset1.bucket_name=vbos-`date "+%Y-%m-%d"` \ --key=vbos_namespace=vizion \ --key=company_name=`echo ${COMPANY_NAME}` \ --key=company_email=`echo ${COMPANY_EMAIL}` \ --key=cluster_uuid=`cat /proc/sys/kernel/random/uuid` err_num=0 if [ "X${SERVER_MODE}" == "Xdev" ]; then cli parameter set --key=cloud_vendor=gce \ --key=installation_method=online \ --key=default_registry_fqdn=registry.vizion.ai \ --key=gce.disks.monitor_volume.size=100 \ --key=gce.disks.cmap_mc_volume.size=20 \ --key=gce.disks.multiple_volume.k128.cmap_dc_volume.size=100 \ --key=gce.disks.mc_volume.size=20 \ --key=gce.disks.multiple_volume.k128.dc_volume.size=100 \ --key=gce.disks.service_mc_volumes.0.size=20 \ --key=gce.disks.service_pxs_volumes.0.size=100 \ --key=gce.disks.service_storage_cache_volumes.0.size=100 \ --key=gce.disks.j_volumes.0.size=100 \ --key=vset.type.vset1=volume,s3 \ --key=vset.type.vset1groupcmapdc128kdplid=1 \ --key=vset.type.vset1groupcmapmcdplid=1 \ --key=vset.type.vset1groupdjcache128kdplid=1 \ --key=vset.type.vset1groupservicedplid=1 \ --key=vset.type.vset1groupjddplid=1 \ --key=vset.type.vset1grouppxsid=1 \ --key=nfs.backend.ext4.vset1.storageSize=2048Gi \ --key=nfs.backend.ext4.vset1.storageClassName=csi-ext4-sc-vset1 err_num=$? else ansible-playbook -i /root/setup/cfg/hosts.ini -e @/root/setup/cfg/custom.yml -v preset-vbos.yml err_num=$? fi if [ ${err_num} -ne 0 ]; then echo "Run vbos preset failed." exit 1 fi ansible-playbook -i /root/setup/cfg/hosts.ini -e @/root/setup/cfg/custom.yml -v vizion-vbos.yml if [ $? -ne 0 ]; then echo "Run vbos setup filed." exit 1 fi exit 0 volumeMounts: - name: vbos mountPath: /etc/kubernetes/vizion volumes: - hostPath: path: /var/log/vbos type: "" name: vbos restartPolicy: Never tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule |
- Run the following command on the bastion host to begin setup:
|
Kubectl apply -f <vbos job yaml> |
4. Confirm that VBOS has been setup properly.
- All pods should be in either the “Running” or “Completed” states.
- You can check by running the command “kubectl get pod” on the bastion host.
- Storage classes “csi-ext4-sc-vset1” and “nfs-provisioner-vset1” should be present, and usable on your apps.
5. Run the provided nginx-deployment-ext4.yam file to test your VBOS deployment.
- After startup, you can use VBOS as a web server.
- Add your index.html file to the root directory of the pod: /usr/share/nginx/html
- Access the web service using: http://<node_ip>:<node_port>
- Below is an example yaml file:
|
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: vbos-nginx-ext4-pvc namespace: vizion spec: accessModes: - ReadWriteOnce resources: requests: storage: 100G storageClassName: csi-ext4-sc-vset1 # our default storageclass volumeMode: Filesystem --- apiVersion: apps/v1 kind: Deployment metadata: labels: name: nginx-ext4 name: nginx-ext4 namespace: vizion spec: replicas: 1 selector: matchLabels: name: nginx-ext4 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: name: nginx-ext4 spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent tty: true volumeMounts: - mountPath: "/usr/share/nginx/html" name: nginx-storage ports: - containerPort: 80 name: "http-server" volumes: - name: nginx-storage persistentVolumeClaim: claimName: vbos-nginx-ext4-pvc nodeSelector: bd: "true" --- kind: Service apiVersion: v1 metadata: name: nginx-ext4 namespace: vizion spec: selector: name: nginx type: ClusterIP ports: - name: p80 port: 80 protocol: TCP targetPort: 80 |