Ingress traffic#

This section describes the architecture and choices made around ingress traffic on Caascad and NGOT product.

Definition and services covered#

Ingress traffic is the traffic coming into our network. We designed a complex solution offering those services:

Our services are exposed through fixed IPS (achieved by loadbalancers and floating ip)
Our services can be DNS resolved through stables FQDN
Our services may use network ACL block traffic from unwanted networks
Applications are able to expose services to ingress traffic with SSL enabled

Technologies#

The technologies involved here are:

FE loadbalancers v2 and v3
FE listener v1 and v2
AWS NLB
NGINX ingress controller
cert-manager
external-dns

Configurations assets#

Assets used to configure services:

FE loadbalancer#

FE loadbalancer terraform configuration

The terraform configuration creates FE Shared Loadbalancers with the FE loadbalancer V2 api, assigns floating IPs to them and then creates a A DNS record with the public IP. It is used only on 1.19 clusters because our envs-ng code is tightly linked to the ingress_controller configuration which is deprecated for kubernetes > 1.19.

Trackbone I/O:
- Trackbone Inputs: N/A
- Trackbone Outputs: loadbalancer ID

envs-ng attributes:

envs: [string]: configurations: fe_loadbalancer: _ // CCE version: 1.19

Contexts: caascad, ngot, pf

Other attributes in the configuration file reflects the terraform resource api.

FE loadbalancer v3#

FE loadbalancer v3 terraform configuration

The terraform configuration creates FE Dedicated Loadbalancers with the FE loadbalancer V3 api, assigns floating ip and then creates multiple DNS records with both public and private ips.

Trackbone I/O:
- Trackbone Inputs: DNS zones map
- Trackbone Outputs: loadbalancer ID, public and private ips, port id

envs-ng attributes:

envs: [string]: configurations: fe_loadbalancer: _ // CCE version: 1.19

Contexts: caascad, ngot, pf

Ingress controller#

nginx-ingress helm chart

A helm chart to deploy NGINX ingress controller using the online nginx-ingress chart. It is compatible with CCE<=1.19. It uses FE loadbalancer ouput to add an annnotation to the ingress service object leading CCE to populate loadbalancers backend with kubernetes nodes containing nginx pods.

Trackbone I/O:
- Trackbone Inputs: FE loadbalancer ID
- Trackbone Outputs: N/A
envs-ng attributes:

envs: [string]: configurations: ingress_controller_private: _ 
envs: [string]: configurations: ingress_controller_public: _

Contexts: caascad, pf

Ingress controller v2#

ingress-nginx helm chart

Same as before but with the ingress-nginx online chart. The chart is compatible with CCE >= 1.21.

Trackbone I/O:
- Trackbone Inputs: FE loadbalancer ID
- Trackbone Outputs: ingress controller pods ips and ports
envs-ng attributes:

envs: [string]: configurations: ingress_controller_private: _ 
envs: [string]: configurations: ingress_controller_public: _

Contexts: caascad, pf, ngot

FE whitelist#

FE whitelist terraform configuration

A terraform configuration which lists all loadbalancers and works on loadbalancers name given as input. It will apply the v1 API whitelist version.

Trackbone I/O:
- Trackbone Inputs: N/A
- Trackbone Outputs: N/A
envs-ng attributes:

envs: [string]: configurations: ingress_controller_v2_private: _ 
envs: [string]: configurations: ingress_controller_v2_public: _

Contexts: caascad, pf, ngot

cert-manager#

cert-manager helm chart

A helm chart that uses cert-manager online helm chart to deploy cert manager. It is responsible for providing certificates object for every certificate or ingress object in the API.

Trackbone I/O:
- Trackbone Inputs: N/A
- Trackbone Outputs: N/A
envs-ng attributes:

envs: [string]: configurations: cert-manager: _

Contexts: caascad, pf, ngot

external-dns#

external-dns helm chart

A helm chart which deploys external-dns binary. It is reponsible of DNS records creation for every ingress objects in the cluster

Trackbone I/O:
- Trackbone Inputs: N/A
- Trackbone Outputs: N/A
envs-ng attributes:

envs: [string]: configurations: external_dns: _

Contexts: caascad, pf, ngot

FE listener#

FE listener terraform configuration

A terraform configuration offering a different way to secure loadbalancers listeners. It searches for loadbalancers and add them listeners. For each listener an ip address group is created and attached to the listener as an ip whitelist container.

Trackbone I/O:
- Trackbone Inputs: ingress controller ips and ports
- Trackbone Outputs: N/A
envs-ng attributes:

envs: [string]: configurations: fe_listener: _

Contexts: caascad, pf, ngot

Workflows#

Here we will see the different provisioning workflows of the previous configuration assets in the old and new styles. The pf context uses caascad style provisioning

Old style provisioning#

fe_base configurations which creates the dns zone used for ingress traffic. fe_base outputs the zone id created. (for NGOT, fe_base takes it from ngot_dns_zones configuration).
fe_loadbalancer takes the zone id, and a basic map declaring loadbalanacers as inputs. It creates loadbalancers and elastic ips and outputs loadbalancer id
ingress_controller takes the suffix from its own name in envs-ng and uses it as ingress class (ingress_controller_suffix). It takes the loadbalancer id and put it in annotation of the ingress controller service object. Finally it also defines a publish-status-address which is used by external-dns to alter the ingresses objects addresses.
When the chart is deployed, CCE will find the annotation with the loadbalancer id and ips/ports of ingress controller pods and creates corresponding listeners on the loadbalancer. It also puts private and public loadbalancer ip to the ingress controller service address
fe_whitelist will find the loadbalancer and apply ACL on its listener according to the acl defined in fe_whitelist.cue

ex:

the dns zone created is ocb-test06.caascad.com
the loadbalancer created is named public, its FQDN is public.ocb-test06.caascad.com
the ingress controller configuration is named ingress_controller_public
the ingress class is public
the address of an ingress object binding the ingress class is public.ocb-test06.caascad.com

 kubectl get ingress -A
NAMESPACE           NAME                                            CLASS                   HOSTS                                             ADDRESS                          PORTS     AGE
cattle-system       rancher                                         <none>                  rancher.ocb-test06.caascad.com                    public.ocb-test06.caascad.com    80, 443   252d
clair               clair                                           <none>                  clair.ocb-test06.caascad.com                      public.ocb-test06.caascad.com

New style provisioning#

fe_base or ngot_dns_zones configurations which will create the dns zones used for ingress traffic. It outputs the zone id created. For NGOT, fe_base takes it from ngot_dns_zones configuration. Please note ngot_dns_zone will manage 2 dns zones, one for the loadbalancer public ip, and the other one for the private ip.
fe_loadbalancer_v3 takes a map containing for each loadbalancers the zone ids where it will have to create public and private dns record with public or private ips
ingress_controller_v2 takes the suffix from its own name in envs-ng and uses it as ingress class (ingress_controller_suffix). It takes the loadbalancer id and put it in annotation of the ingress controller service object. Finally it also defines a publish-status-address which is used by external-dns to alter the ingresses objects addresses.
When the chart is deployed, CCE will find the annotation with the loadbalancer id and ips/ports of ingress controller pods and creates corresponding listeners on the loadbalancer. It also puts private and public loadbalancer ip to the ingress controller service address
fe_whitelist will find the loadbalancer and apply ACL on its listener according to the acl defined in fe_whitelist.cue

FE_listener provisioning style#

As previously fe_base or ngot_dns_zones will take care of dns zones provisioning
As previously fe_loadbalancer_v3 will manage loadbalancers, eips, and dns records
ingress_controller_v2 is deployed without annotating the service object with the loadbalancer id. Instead the service is declared as a NodePort service to get fixed ips and ports for the loadbalancer.
As the service object is not annotated, CCE will do nothing with LB(s)
fe_listener will take a map declaring listeners to create on the loadbalancer with ips and ports given by ingress_controller_v2. For each element in the map it will create a listener on the LB, an ip group object and assign it to the listener as a whitelist container.

Ingress lifecycle#

A random chart creates an ingress object named and bound to the public ingress class
Public ingress controller find the new ingress and the corresponding service and builds a new server directive with the service as backend. The controller updates the ingress address field with the publish-status-address helm attributes previously built in envs-ng
External-dns find the new ingress object and creates a new record with the ingress address field. Because this field is a fqdn, external dns will build a cname:
```
myingress IN CNAME public.<zone name>
```
Cert-manager find the new ingress object, and starts to issue a certificate for myingress.
Ingress controller finds the certificate secrets created by cert-manager and uses it for SSL termination

Caveheats#

Whitelists generated by fe_whitelists may disappear. Whitelists generated by fe_whitelist are using FE listener v1 API. The service allows us to define a string containing all ip addresses whitelisted or blacklisted. CCE offers no way to fill that field through the service object as we do with the loadbalancer id. Thus the old style provisioning workflow requires to apply whitelists after the listener has been created by CCE. It results in CCE manages emptying that field every time it will provision a new listener (in a destroy and apply scenario for example).

Due to this security breach a careful attention should be payed over fe_whitelists drift in CI daily reports.

Another consequence is that it is hard to split whitelists provisioning among configurations. The API limitation force us to fill the whitelist fully on a single configuration run.

To overcome those issues a new provisioning style was provided with the FE_listener provisioning style. With this method, we do not allow CCE to manage listeners for us. Instead we build them ourselves with the FE listener v2 API. This version of the API let us add an IP group object ID to the listener when we create it. This IP group should pre exist and may already be fed by allowed IPs. So with this method we can guarantee that when a listener is provisionned, it is already fully secured.

Architecture decision#

The ingress controller should be the only deployment declaring a loadbalancer type service object in kubernetes. All other services should declare ClusterIP type service objects.
AWS auto creates loadbalancers for every Loadbalancer type service object in kubernetes. FinOPS consideration as well as EIPs quotas on AWS lead us to prevent loadbalancers multiplication. So we decided to change every loadbalancer service object in a ClusterIP service object. This way those services became nginx ingress backends, and we limit the number of loadbalancers to the number of ingress controllers.
The ingress controller publish-status-address helm chart attribute should be filled with the loadbalancer FQDN.
When an ingress controller sees a new ingress object, if the ingress class matches, it will build a backend for it and update the ingress address field with the loadbalancer IPs.
External-dns will use that address field to create a A type dns record.
At some point CCE started to publish both public and private loadbalancers IPs on kubernetes loadbalancer type services instead of the public ip only. External-dns acted weirdly and tried to add records for both ips in the same zone, leading to connectivity issues.
To overcome that situation, we used the publish-status-address attribute. With the publish-status-address we force the ingress controller to put a specific value. We used the attribute to force the ingress controller to write the loadbalancer FQDN, instead of its ips.
When external-dns sees a FQDN in the ingress object address, it creates a CNAME type records:
```
myingress IN CNAME public.ocb-test06.caascad.com
```
Finally we added loadbalancers dns records creation in the fe_loadbalancer configuration.
```
public IN A 1.2.3.4
```