Calico Networking with Tanzu Community Edition Part 1
Updated: Jan 18, 2022
When you are first starting out with Kubernetes, networking is probably one of the most difficult concepts to master and understand. There are many choices and decisions to be made, and understanding the pros and cons of each can be bewildering.
Pods and containers need to be able to communicate with each other internally, externally and to/from the outside world. Kubernetes does a good job of obscuring many of these complexities by making things more "public cloud" like (i.e. by hiding many of the details from the admin that way you can focus more on your apps and less on the cloud platform itself).
But modern applications demand enterprise grade, advanced networking and security services. In order to manage networking at scale and offer a degree of granular control, a technology has been introduced: The Container Network Interface (CNI).
From the official CNCF documentation:
"CNI (Container Network Interface), a Cloud Native Computing Foundation project, consists of a specification and libraries for writing plugins to configure network interfaces in Linux containers, along with a number of supported plugins."
There are a number of CNI plugins available, each with there own features and use cases.
In 2020 ITNEXT published an article detailing the most commonly used CNI`s each with there performance ratings and feature descriptions. An excellent summary is shown here:
CNI`s provide a degree of networking standardization across the ecosystem.
In this article I wanted to explain my understanding of the Calico plugin available from Tigera, and show an example of what this looks like when deployed in a TCE (with vSphere) environment.
I am currently running TCE on a vSphere 6.7U3 lab environment with vSAN and NSX-T networking. Calico will work with almost any underlying networking infrastructure, as long as there is network reach-ability between the nodes.
1.) Install Calico Package
Create a TCE repo:
tanzu package repository add tce-repo \ --url projects.registry.vmware.com/tce/main:0.9.1 \ --namespace tanzu-package-repo-global
Use the following command to install the Calico package:
tanzu package install calico-pkg --package-name calico.tanzu.vmware.com --namespace tkg-system --version 3.11.3+vmware.1-tkg.1 --wait=false
tanzu package available get calico.tanzu.vmware.com/3.11.3+vmware.1-tkg.1 --namespace tkg-system
2.) Create cluster configuration file
Create a workload cluster configuration file, and save it in the following directory. This is the default cluster config file location for TCE:
An example config file can be found here:
AVI_CA_DATA_B64: "" AVI_CLOUD_NAME: "" AVI_CONTROL_PLANE_HA_PROVIDER: "" AVI_CONTROLLER: "" AVI_DATA_NETWORK: "" AVI_DATA_NETWORK_CIDR: "" AVI_ENABLE: "false" AVI_LABELS: "" AVI_MANAGEMENT_CLUSTER_VIP_NETWORK_CIDR: "" AVI_MANAGEMENT_CLUSTER_VIP_NETWORK_NAME: "" AVI_PASSWORD: "" AVI_SERVICE_ENGINE_GROUP: "" AVI_USERNAME: "" CLUSTER_CIDR: 192.168.0.0/11 CLUSTER_NAME: tce-prod-01 CLUSTER_PLAN: prod ENABLE_AUDIT_LOGGING: "false" ENABLE_CEIP_PARTICIPATION: "false" ENABLE_MHC: "true" IDENTITY_MANAGEMENT_TYPE: none INFRASTRUCTURE_PROVIDER: vsphere LDAP_BIND_DN: "" LDAP_BIND_PASSWORD: "" LDAP_GROUP_SEARCH_BASE_DN: "" LDAP_GROUP_SEARCH_FILTER: "" LDAP_GROUP_SEARCH_GROUP_ATTRIBUTE: "" LDAP_GROUP_SEARCH_NAME_ATTRIBUTE: cn LDAP_GROUP_SEARCH_USER_ATTRIBUTE: DN LDAP_HOST: "" LDAP_ROOT_CA_DATA_B64: "" LDAP_USER_SEARCH_BASE_DN: "" LDAP_USER_SEARCH_FILTER: "" LDAP_USER_SEARCH_NAME_ATTRIBUTE: "" LDAP_USER_SEARCH_USERNAME: userPrincipalName OIDC_IDENTITY_PROVIDER_CLIENT_ID: "" OIDC_IDENTITY_PROVIDER_CLIENT_SECRET: "" OIDC_IDENTITY_PROVIDER_GROUPS_CLAIM: "" OIDC_IDENTITY_PROVIDER_ISSUER_URL: "" OIDC_IDENTITY_PROVIDER_NAME: "" OIDC_IDENTITY_PROVIDER_SCOPES: "" OIDC_IDENTITY_PROVIDER_USERNAME_CLAIM: "" OS_ARCH: amd64 OS_NAME: ubuntu OS_VERSION: "20.04" SERVICE_CIDR: 192.168.0.0/13 TKG_HTTP_PROXY_ENABLED: "false" VSPHERE_CONTROL_PLANE_DISK_GIB: "20" VSPHERE_CONTROL_PLANE_ENDPOINT: VSPHERE_CONTROL_PLANE_MEM_MIB: "4096" VSPHERE_CONTROL_PLANE_NUM_CPUS: "2" VSPHERE_DATACENTER: VSPHERE_DATASTORE: VSPHERE_FOLDER: VSPHERE_NETWORK: VSPHERE_PASSWORD: VSPHERE_RESOURCE_POOL: VSPHERE_SERVER: VSPHERE_SSH_AUTHORIZED_KEY: VSPHERE_TLS_THUMBPRINT: VSPHERE_USERNAME: VSPHERE_WORKER_DISK_GIB: "20" VSPHERE_WORKER_MEM_MIB: "4096" VSPHERE_WORKER_NUM_CPUS: "2"
3.) Deploy cluster with Calico CNI:
CNI`s cannot be replaced once a cluster is deployed, because it provides the essential networking services for the cluster. Antrea is the default CNI when another is not specified.
The CNI must be selected when creating the cluster.
tanzu cluster create --file tce-prod-test.yaml --cni=calico
The deployment would have been created now, use the following commands to set the kubectl context:
tanzu cluster kubeconfig get tce-prod-test --admin kubectl config use-context tce-prod-test-admin@tce-prod-test
4.) Install calicoctl
Most calicoctl commands require access to the Calico datastore. By default, calicoctl will attempt to read from the Kubernetes API based on the default kubeconfig. This allows the Calico control plane to have visibility into the Kubernetes environment.
There are also two ways of running calicoctl: either by installing it as a binary or by installing it as a kubectl plugin. I have chosen to install the kubectl plugin version. The only caveat here is that you use "kubectl calico" instead of the calicoctl syntax.
Note: If the location of calicoctl is not already in your PATH, move the file to one that is or add its location to your PATH. This will allow you to invoke it without having to prepend its location.
Find your path:
curl -o kubectl-calico -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.11.3/calicoctl"
Make the file executable:
chmod +x kubectl-calico
kubectl calico -h
5.) Locate your kubeconfig file
Run the following command to locate your kubeconfig file:
[[ ! -z "$KUBECONFIG" ]] && echo "$KUBECONFIG" || echo "$HOME/.kube/config"
This will return the kubeconfig location:
6.) Create configuration file
Specify the kubeconfig file location and datastore type:
apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: datastoreType: "kubernetes" kubeconfig: "/home/ice/.kube/config"
Save the file and exit.
7.) Export configurations
export DATASTORE_TYPE=kubernetes export KUBECONFIG=~/.kube/config
8.) Run test command
Run a kubectl calico command to test:
kubectl calico ipam show
You can now run any calicoctl subcommands through kubectl calico.