Troubleshooting HTTP Proxy Connectivity Issues with VKS on VCF 9.0.1

Many enterprise data centers do not allow direct internet access from the internal network for security reasons. An HTTP proxy allows VCF components (like SDDC Manager) and VKS clusters to download necessary patches, update bundles, and container images from external online repositories or private registries securely.

An HTTP proxy for VMware vSphere Kubernetes Service (VKS) on VMware Cloud Foundation (VCF) is primarily used to provide network isolation and security, enabling communication between components in isolated networks (like management and workload networks) and allowing internet access for updates and logging in a controlled manner

Repo images for VKS, such as ArgoCD are pulled from ESX (via workload network). Pods that get spun up from these services utilize the NodePort networking construct in Kubernetes.

A NodePort exposes a Service on a static port on each cluster node’s IP address, allowing external traffic to access the service through <NodeIP>:<NodePort>.

Kubernetes assigns the same port on every node, chosen from a range specified by the --

service-node-port-range flag (default is 30000 through 32767). You define the specific port in the Service resource configuration

NodePort Concept in Kubernetes

Instructions on how to configure a proxy for your VKS environment can be found here: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/vsphere-supervisor-installation-and-configuration/configuring-and-managing-a-supervisor-cluster/configuring-http-proxy-settings-in-vsphere-with-tanzu.html

Further details on how the networking is setup under the hood for this can be found here: https://knowledge.broadcom.com/external/article/410597/networking-requirements-for-downloading.html

There might be a situation where you have deployed an external proxy and you have it configured to pull images from projects.packages.broadcom.com but you notice that even though you can curl that URL from the Supervisor, the images never pull over the network and go into a running state. When describing these failed Pods you see the following errors:

Normal Pulling 2m9s (x2 over 4m11s) kubelet Waiting for Image svc-argocd-service-domain-c10/argocd-service-76397c8fb4ac94a3699cb8009b68f8d1997bf192-v45252

Normal Image 2m8s image-controller Image argocd-service-76397c8fb4ac94a3699cb8009b68f8d1997bf192-v43402 bound successfully

Normal Pulling 6s (x2 over 2m8s) kubelet Waiting for Image svc-argocd-service-domain-c10/argocd-service-76397c8fb4ac94a3699cb8009b68f8d1997bf192-v43402

Warning Failed 6s (x2 over 6s) kubelet failed to get images: Image svc-argocd-service-domain-c10/argocd-service-76397c8fb4ac94a3699cb8009b68f8d1997bf192-v43402 has failed. Error: Failed to resolve on node vcf01-w01-esx001.vrack.vcf.com. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head "https://projects.packages.broadcom.com/v2/vsphere/supervisor/argocd-service/1.0.0/argocd-service/ma… proxyconnect tcp: dial tcp YOUR_PROXY_IP:8080: connect: network is unreachable

This indicates that there is some kind of networking or proxy issue. The first step here is to make sure that your workload network (Overlay) in NSX can reach the external world via the Tier0 Provider Gateway cluster. Ensure that the workload network and any auto configured static routes (SNAT) are being advertised over BGP: https://knowledge.broadcom.com/external/article/339392/troubleshooting-bgp-on-nsxt-edge-nodes.html

If that works, there might be an issue internally in VKS. Through troubleshooting this week with one of my favorite customers, I compiled a list of diagnostic commands that might aid others on the quest to get this to work.

SSH to the Supervisor as root and run the following diagnostics:

######################################################

Check kappcontroller, this is where Proxy settings are inherited from vCenter and then get injected into VKS:

kubectl get secret kapp-controller-config -n vmware-system-appplatform-operator-system -o yaml

Look for the noProxy section and decode this:

echo "your_random_binary_base64_string" | base64 -d

You should see proxy servers listed here as well as NO_PROXY configurations, if these appear in the list that means that proxy settings were applied correctly

######################################################

Check that the kapp controller can reach the proxy:

Access the kappcontroller:

kubectl exec -it kapp-controller -n vmware-system-appplatform-operator-system -- /bin/sh

Run diagnostics:

kapp-controller: ping projects.packages.broadcom.com <--- If this is not reachable, then routing from Proxy to outside world does not work

kapp-controller: ping YOUR_PROXY_IP <--- If this is reachable, then routing to Proxy works

kapp-controller: curl -vk https://projects.packages.broadcom.com/v2/ <--- If this is reachable, then routing from K8s to outside world works

######################################################

Check DNS from Supervisor:

dig -b YOUR_DNS_IP projects.packages.broadcom.com <--- The -b option sets the source address 

######################################################

Test Proxy reachability from Supervisor:

tcpdump -i eth1 host YOUR_PROXY_IP <--- Workload Network

tcpdump -i eth0 host YOUR_PROXY_IP <--- Management Network 

######################################################

Check that Proxy itself has reachability to the following:

projects.registry.vmware.com

projects.packages.broadcom.com

######################################################

Check that proxy settings are applied in /etc/containerd/econtained-environment:

grep -Erin 'proxy|dns' /etc/containerd/

This will validate that the kappcontroller has successfully pushed proxy settings to Kubernetes.

######################################################

Disclaimer: This is my personal blog. The opinions expressed here represent my own and not those of the vendors, brands or my employer.