Skip to main content

Configure Proxy CA Certificates for Workload Clusters

When your proxy server uses a Certificate Authority (CA) certificate for TLS inspection, workload cluster nodes and pods need access to the CA certificate to establish trusted connections through the proxy. The Reach system handles injecting proxy environment variables into pods automatically, but the CA certificate must be configured separately.

Proxy CA certificates provided during PCG installation are propagated to the PCG cluster nodes but are not automatically propagated to workload cluster nodes. You must configure the CA certificate for workload clusters using one of the methods described in this guide.

There are two levels of CA certificate trust to consider:

  • Node-level trust - The CA certificate is written to each workload cluster node's filesystem and added to the operating system trust store. System services on the node, such as containerd and kubelet, will trust the proxy. Both the tenant-level and cluster profile approaches provide node-level trust.

  • Pod-level trust - The CA certificate is mounted into pods as a volume so that applications running inside containers can trust the proxy. Pod-level trust requires the podMount configuration in the cluster profile OS layer and is only available through the cluster profile approach. Certificates configured at the tenant level are not mounted into pods.

Add CA Certificate to Workload Cluster Nodes​

To make the proxy CA certificate available to workload cluster nodes, configure it at either the tenant level or the cluster profile level in the OS layer.

  • Tenant level - All workload clusters provisioned from the tenant, with the exception of managed Kubernetes clusters (EKS, AKS, and GKE) and Edge clusters, have the CA certificate injected into their cluster nodes.

  • Cluster profile OS level - Only workload clusters deployed using the cluster profile have the CA certificate injected.

Tenant Level​

Use this approach to propagate your proxy server CA certificate to all workload cluster nodes provisioned from the tenant, with the exception of managed Kubernetes clusters (EKS, AKS, and GKE) and Edge clusters. For managed Kubernetes and Edge clusters, proceed to the Cluster Profile OS Layer section.

warning

Tenant-level certificates provide node-level trust only. Applications running inside pods do not inherit the node trust store and will not trust the proxy CA certificate. If your workloads need to make HTTPS requests through the proxy, use the Cluster Profile OS Layer approach with the podMount configuration to mount the certificate into pods.

Prerequisites​

Enablement​

  1. Log in to Palette as a tenant admin.

  2. From the left main menu, select Tenant Settings.

  3. From the Tenant Settings menu, below Platform, select Certificates.

  4. Select Add A New Certificate.

  5. In the Add Certificate dialog, enter the Certificate Name and Certificate value.

  6. Confirm your changes.

Validate​

  1. Log in to Palette.

  2. Deploy a workload cluster through using the PCG, or apply the updated cluster profile to an existing workload cluster.

  3. SSH into a node in the workload cluster. Refer to the SSH Keys and SSH Usernames guides for information on accessing cluster nodes.

    ssh -i <private-key-path> <username>@<dns-or-ip-address>
    Example command
    ssh -i ~/.ssh/spectro2026.pem spectro@10.10.149.37

  4. Verify the CA certificate file is present on the node.

    ls -l /usr/local/share/ca-certificates/spectro-ca.crt
    Example output
    -rw-r--r-- 1 root root 1049 Mar  6 13:46 /usr/local/share/ca-certificates/spectro-ca.crt

  5. Verify the certificate entered at either Tenant Settings > Certificates matches the CA certificate on the node.

    cat /usr/local/share/ca-certificates/spectro-ca.crt
    Example output
    -----BEGIN CERTIFICATE-----
    **************************
    -----END CERTIFICATE-----

  6. Verify the certificate is included in the system trust store.

    openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /usr/local/share/ca-certificates/spectro-ca.crt
    Example output
    /usr/local/share/ca-certificates/spectro-ca.crt: OK

Cluster Profile OS Layer​

Use this approach to propagate proxy server CA certificates on a per-cluster basis. This is the only approach that supports mounting the CA certificate into pods through the Reach system.

Prerequisites​

Enablement​

  1. Log in to Palette.

  2. From the left main menu, select Profiles.

  3. Choose an existing cluster profile or create a new cluster profile. For more information, refer to the Cluster Profiles guide.

  4. In the OS layer of your cluster profile, add your CA certificate to the content field under kubeadmconfig.files. The CA certificate file must be named ca.crt.

    By default, adding a proxy CA certificate to the cluster profile OS layer places the certificate on each workload cluster node and adds it to the node trust store. However, applications running inside pods do not automatically inherit the node trust store. To make the CA certificate available inside pods, use the podMount configuration in the kubeadmconfig.files section of the OS layer. The podMount configuration instructs Palette to mount the specified host file into pods through the Reach system.

    The following table describes the configuration fields.

    FieldDescription
    targetPathThe path on the host node where the CA certificate file is written. The certificate file must be named ca.crt.
    targetOwnerThe file ownership in the format user:group.
    targetPermissionsThe file permissions in octal notation.
    contentThe PEM-encoded CA certificate content.
    podMount.allowedSet to true to enable mounting the host file into pods through the Reach system.
    podMount.targetPathThe path inside pods where the file is mounted. Use /etc/ssl/certs/ca-certificates.crt to make the certificate available to most applications that use the system trust store.
    Example OS layer configuration
    kubeadmconfig:
      preKubeadmCommands:
        - echo "Executing pre kube admin config commands"
        - update-ca-certificates
        - "systemctl restart containerd; sleep 3"
        - 'while [ ! -S /var/run/containerd/containerd.sock ]; do echo "Waiting for containerd..."; sleep 1; done'
      postKubeadmCommands:
        - echo "Executing post kube admin config commands"
      files:
        - targetPath: /usr/local/share/ca-certificates/ca.crt
          targetOwner: "root:root"
          targetPermissions: "0644"
          content: |
            -----BEGIN CERTIFICATE-----
            *************************
            -----END CERTIFICATE-----
          podMount:
            allowed: true
            targetPath: /etc/ssl/certs/ca-certificates.crt
    warning

    When using podMount, the Reach system creates a host-path volume from the file at targetPath on the node and mounts it at podMount.targetPath inside pods. If the individual proxy CA certificate file is mounted at /etc/ssl/certs/ca-certificates.crt, it replaces the container's default CA bundle. This means pods will trust your proxy CA but will not trust standard public Certificate Authorities.

    In most proxy environments this is acceptable because all outbound HTTPS traffic passes through the proxy, which handles upstream TLS verification on behalf of the pod. However, if your pods make direct TLS connections to services that bypass the proxy, such as internal services listed in NO_PROXY, those connections may fail because the container no longer has the default public CA certificates.

    To preserve the container's default CA bundle, use a path other than /etc/ssl/certs/ca-certificates.crt for podMount.targetPath. However, doing so requires individual configurations for each app to point to the custom podMount.targetPath.

  5. Select Confirm Updates.

Validate Node-Level Trust​

  1. Log in to Palette.

  2. Deploy a workload cluster through using the PCG, or apply the updated cluster profile to an existing workload cluster.

  3. SSH into a node in the workload cluster. Refer to the SSH Keys and SSH Usernames guides for information on accessing cluster nodes.

    ssh -i <private-key-path> <username>@<dns-or-ip-address>
    Example command
    ssh -i ~/.ssh/spectro2026.pem spectro@10.10.149.37

  4. Verify the CA certificate file is present on the node.

    ls -l /usr/local/share/ca-certificates/ca.crt
    Example output
    -rw-r--r-- 1 root root 1049 Mar  6 13:46 /usr/local/share/ca-certificates/ca.crt

  5. Verify the certificate entered in the cluster profile OS layer matches the CA certificate on the node.

    cat /usr/local/share/ca-certificates/ca.crt
    Example output
    -----BEGIN CERTIFICATE-----
    **************************
    -----END CERTIFICATE-----

  6. Verify the certificate is included in the system trust store.

    openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /usr/local/share/ca-certificates/ca.crt
    Example output
    /usr/local/share/ca-certificates/ca.crt: OK

Validate Pod-Level Trust​

  1. Log in to Palette.

  2. Deploy a workload cluster through using the PCG, or apply the updated cluster profile to an existing workload cluster.

  3. Download your workload cluster's kubeconfig and configure kubectl to access the cluster. Refer to our Access Cluster with CLI guide for more information.

  4. Verify the ClusterPodPreset contains volume and volumeMount entries for the CA certificate.

    kubectl get clusterpodpreset proxy --output yaml | grep --after-context=10 volumeMounts

    The output should include volumes and volumeMounts entries referencing the CA certificate path.

    Example output
    volumeMounts:
      - metadata:
          merge_strategy: replace
        mountPath: /etc/ssl/certs/ca-certificates.crt
        name: ca-crt-11d45c1c
    volumes:
      - hostPath:
          path: /usr/local/share/ca-certificates/ca.crt
          type: File
        metadata:
          merge_strategy: replace

  5. Create a temporary namespace with the privileged Pod Security Standard. The Reach system injects hostPath volumes into matching pods, which requires the privileged security level.

    kubectl create namespace test-proxy-cert
    kubectl label namespace test-proxy-cert pod-security.kubernetes.io/enforce=privileged

  6. Deploy a test pod with the spectrocloud.com/connection: proxy label. The Reach system only injects volumes into pods that match the ClusterPodPreset's label selector.

    kubectl run test-proxy-cert --namespace test-proxy-cert --image=busybox --restart=Never \
      --labels="spectrocloud.com/connection=proxy" \
      --command -- sleep 3600

  7. Verify the certificate file is mounted inside the container. The certificate should match the content pasted in to the cluster profile OS layer.

    kubectl exec test-proxy-cert --namespace test-proxy-cert -- cat /etc/ssl/certs/ca-certificates.crt
    Example output
    -----BEGIN CERTIFICATE-----
    **************************
    -----END CERTIFICATE-----

  8. Clean up the test namespace and pod.

    kubectl delete namespace test-proxy-cert