WAF - ModSecurity

Mise en place de ModSecurity sur Ingress.

Il y a 2 façon de déployer ModSecurity sur un Ingress Nginx. La plus simple est la plus logique est d’ajouter la configuration en annotation sur l’Ingress. Cette solution permet d'adapter la configuration pour chaque Ingress, donc pour chaque Services.

Cependant au delà d’une quinzaine d’ingress, le temps de chargement du controller est trop long (prévoir ~8 minutes pour 22 Ingress), ce qui provoque des conflits avec le admissionWebhooks, provoque des restart des controller, bloque les déploiements helm.

L'autre solution, qui n’est pas franchement documenté, c'est de configurer ModSecurity au niveau du controller directement. C’est beaucoup plus performant mais cela touche toutes les connexions au nginx (également les api d’autoconfiguration, de metrics, etc..)

L’inconvénient de cette solution est qu’il n’est pas possible de fournir une configuration spécifique pour chaque Ingress, ni de désactiver le WAF pour un Ingress en particulier. (Ou alors il faut définir une règle ModSecurity spécifique dans le controller.)

Solution A - Configuration de ModSecurity sur un Ingress

Pour activer le WAF il suffit de modifier la configuration de l'ingress en ajoutant 2 annotations.

• nginx.ingress.kubernetes.io/enable-modsecurity: "true"
• nginx.ingress.kubernetes.io/modsecurity-snippet: ...

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: svc-waf-ingress
  annotations:
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      # OWASP Core Rule Set
      Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
      SecRuleEngine On
      SecRequestBodyAccess On
      SecAuditEngine RelevantOnly
      SecAuditLogParts ABFHIJZ
      SecAuditLogType Concurrent
      SecAuditLogFormat JSON
      SecAuditLog /var/log/modsec/audit.log
      SecAuditLogStorageDir /var/log/modsec/audit/
spec:
  ingressClassName: ingress-nginx-public

La configuration du waf doit être appliquée sur chaque ingress indépendamment, mais un problème de configuration fera planter tous le controller, donc tous les ingress !

Documentation : https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#modsecurity

Solution B - Configuration de ModSecurity sur le Controller.

Pour activer le WAF au niveau du Controller nginx, il faut ajouter 2 paramètres dans le ConfigMap enable-modsecurity et modsecurity-snippet.

Cette configuration est incompatible avec la solution A. Une fois en place il n’est plus possible de désactiver le WAF sur un location/ingress spécifique (contrairement à ce qui est écrit dans la documentation, mais je n'ai pas réussis à le faire.).

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-v2-bastion-controller
  namespace: ingress-nginx-v2-bastion
data:
  enable-modsecurity: "true"
  modsecurity-snippet: |-
    Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
    SecRuleEngine DetectionOnly
    SecRequestBodyAccess On
    SecAuditEngine RelevantOnly
    SecAuditLogParts ABFHIJZ
    SecAuditLogType Concurrent
    SecAuditLogFormat JSON
    SecAuditLog /var/log/modsec/audit.log
    SecAuditLogStorageDir /var/log/modsec/audit/
    # Dynamic reconfiguration failed https://github.com/kubernetes/ingress-nginx/issues/8137
    SecRule REQUEST_HEADERS:Host "@streq 127.0.0.1:10246" "id:21029, phase:1, t:none, nolog, pass, ctl:ruleEngine=Off"

La dernière ligne de configuration est indispensable SecRule REQUEST_HEADERS:Host ... elle permet au controller d’appeler les api d’autoconfiguration sur le port local 10246

Documentation: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-modsecurity

Configuration ModSecurity

Globalement il ne faut pas toucher aux paramètres car ils sont liés à la configuration de Promtail. Cependant on va définir SecRuleEngine DetectionOnly, le temps de mettre en place des règles fines sur l'application ciblée.

• SecRuleEngine
  • On: process rules
  • Off: do not process rules
  • DetectionOnly: process rules but never executes any disruptive actions (block, deny, drop, allow, proxy and redirect)
• SecRequestBodyAccess: This directive is required if you want to inspect the data transported request bodies (e.g., POST parameters). Request buffering is also required in order to make reliable blocking possible.
• SecAuditEngine
  • On: log all transactions
  • Off: do not log any transactions
  • RelevantOnly: only the log transactions that have triggered a warning or an error, or have a status code that is considered to be relevant (as determined by the SecAuditLogRelevantStatus directive)

• SecAuditLogParts = ABFHIJZ

  • A: Audit log header (mandatory).
  • B: Request headers.
  • C: Request body (present only if the request body exists and ModSecurity is configured to intercept it. This would require SecRequestBodyAccess to be set to on).
  • D: Reserved for intermediary response headers; not implemented yet.
  • E: Intermediary response body (present only if ModSecurity is configured to intercept response bodies, and if the audit log engine is configured to record it. Intercepting response bodies requires SecResponseBodyAccess to be enabled). Intermediary response body is the same as the actual response body unless ModSecurity intercepts the intermediary response body, in which case the actual response body will contain the error message (either the Apache default error message, or the ErrorDocument page).
  • F: Final response headers (excluding the Date and Server headers, which are always added by Apache in the late stage of content delivery).
  • G: Reserved for the actual response body; not implemented yet.
  • H: Audit log trailer.
  • I: This part is a replacement for part C. It will log the same data as C in all cases except when multipart/form-data encoding in used. In this case, it will log a fake application/x-www-form-urlencoded body that contains the information about parameters but not about the files. This is handy if you don’t want to have (often large) files stored in your audit logs.
  • J: This part contains information about the files uploaded using multipart/form-data encoding.
  • K: This part contains a full list of every rule that matched (one per line) in the order they were matched. The rules are fully qualified and will thus show inherited actions and default operators. Supported as of v2.5.0.
  • Z: Final boundary, signifies the end of the entry (mandatory).

• SecAuditLogType

  • Concurrent pour des raisons de performance, on active le mode, Concurrent ce qui provoque un fichier de log par requetes.
• SecAuditLogFormat
  • JSON on choisi le format de log Json qui est plus simple a manipuler avec promtail et loki
• SecAuditLog
  • /var/log/modsec/audit.log en placement du log principal d'audit, ca ressemble à un access log qui liste des fichier de log audit. Le dossier /var/log/modsec/ est un volume persistant.
• SecAuditLogStorageDir
  • /var/log/modsec/audit/ Dossier contenant les logs d'audit, il est surveillé par promtail

OWASP Core Rule Set

Par défaut on active les règles OWASP CRS PF1, il peut être nécessaire d'affiner pour chaque application ces règles, soit pour durcir, soit pour assouplir les règles. (En particulier sur le filtrage des commandes HTTP autorisés (allowed_methods))

• https://coreruleset.org/
• https://github.com/SpiderLabs/owasp-modsecurity-crs
• https://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project

Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf -> charge les règles OWASP Core Rule Set ce qui correspond à

Include /etc/nginx/owasp-modsecurity-crs/crs-setup.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-901-INITIALIZATION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-903.9001-DRUPAL-EXCLUSION-RULES.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-903.9002-WORDPRESS-EXCLUSION-RULES.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-905-COMMON-EXCEPTIONS.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-910-IP-REPUTATION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-911-METHOD-ENFORCEMENT.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-912-DOS-PROTECTION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-913-SCANNER-DETECTION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-921-PROTOCOL-ATTACK.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-922-MULTIPART-ATTACK.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-934-APPLICATION-ATTACK-NODEJS.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-944-APPLICATION-ATTACK-JAVA.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-950-DATA-LEAKAGES.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-959-BLOCKING-EVALUATION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf
Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

Configuration Standard

Le réglage par défaut

• Bloque les requêtes malveillante avec un error 403
• Log les requêtes malveillante
• Paranoia Level = PL1 (Le niveau le plus faible)
• Blocking Threshold Levels
  • inbound_anomaly_score_threshold=5
  • outbound_anomaly_score_threshold=4
• HTTP Policy Settings
  • allowed_methods=GET HEAD POST OPTIONS
  • allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/json| |application/cloudevents+json| |application/cloudevents-batch+json|
  • allowed_http_versions=HTTP/1.0 HTTP/1.1 HTTP/2 HTTP/2.0
  • restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .rdb/ .resources/ .resx/ .sql/ .swp/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/
  • restricted_headers=/accept-charset/ /content-encoding/ /proxy/ /lock-token/ /content-range/ /if/
  • static_extensions=/.jpg/ /.jpeg/ /.png/ /.gif/ /.js/ /.css/ /.ico/ /.svg/ /.webp/
  • allowed_request_content_type_charset=utf-8|iso-8859-1|iso-8859-15|windows-1252
• HTTP Argument/Upload Limits
  • max_num_args=255
  • arg_name_length=100
  • arg_length=400
  • total_arg_length=64000
  • max_file_size=1048576
  • combined_file_sizes=1048576

Promtail / Loki

Pour remonter les logs dans Loki, il faut configurer un container promtail sur le pod de l'ingress controller.

1. Configuration d'un pvc pour contenir les audits logs
2. Configuration d'un secret contenant la configuration promtail et les credentials de connexion à Loki
3. Paramétrage de la Chart Helm pour mettre en place le sidecar container promtail

ingress-controller-v2/helm/templates/pvc.yaml

{{- if .Values.pvc.name -}}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{.Values.pvc.name}}
spec:
  accessModes:
    {{- with .Values.pvc.accessModes }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
  storageClassName: {{.Values.pvc.storageClassName}}
  volumeMode: {{.Values.pvc.volumeMode}}
  resources:
    requests:
      storage: {{.Values.pvc.resources.requests.storage}}
{{- end }}

ingress-controller-v2/helm/templates/secret-promtail.yaml

{{- define "_promtail.yaml" -}}
server:
  log_level: {{ .Values.promtail.config.logLevel }}
  http_listen_port: {{ .Values.promtail.config.serverPort }}

{{- if .Values.promtail.config.clients }}
clients:
  {{- toYaml .Values.promtail.config.clients | nindent 6 }}
{{- end }}

positions:
  filename: /var/log/modsec/positions.yaml

scrape_configs:
  - job_name: modsec
    pipeline_stages:
      - json:
          expressions:
            transaction: transaction
            request: transaction.request
            response: transaction.response
            messages: transaction.messages
      - json:
          expressions:
            client_ip: client_ip
            client_port: client_port
            host_ip: host_ip
            host_port: host_port
            time_stamp: time_stamp
            unique_id: unique_id
          source: transaction
      - json:
          expressions:
            method: method
            uri: uri
            host: headers.Host
            headers: headers
          source: request
      - json:
          expressions:
            http_code: http_code
          source: response
      - template:
          source: level
          template: "WARN"
      - template:
          source: app
          template: "${APPNAME}"
      - template:
          source: cluster
          template: "${CLUSTER}"
      - template:
          source: pod
          tempalte: "${HOSTNAME}"
      - template:
          source: unixms
          template: "{{"{{"}} .unique_id  | substr 0 12 }}0"
      - labels:
          app:
          cluster:
          host:
          level:
          client_ip:
          client_port:
          host_ip:
          host_port:
          method:
          http_code:
      - timestamp:
          format: UnixMs
          source: unixms
    static_configs:
      - targets:
          - localhost
        labels:
          job: modsecaudit
          __path__: {{ .Values.promtail.config.modsecAuditDir }}/*/*/*
{{- end }}

{{- if .Values.promtail.config -}}
---
apiVersion: v1
kind: Secret
metadata:
  name: promtail-config
type: Opaque
data:
  promtail.yaml: {{ include "_promtail.yaml" . | b64enc }}
{{- end }}

si vous changez le secret, le controller ne se redéploie pas automatiquement, il faut faire un rollout.

kubectl rollout restart -n ingress-nginx-v2-public deployment ingress-nginx-v2-public-controller

contexts/ingress_controller_v2.cue

extraContainers: [
  {
    name:  "promtail"
    image: "grafana/promtail"
    args: [
      "-config.file=/etc/promtail.yaml",
      "-config.expand-env=true",
    ]
    env: [
      {name: "CLUSTER", value: zone.name},
      {name: "APPNAME", value: ConfigName},
    ]
    volumeMounts: [
      {
        name:      "config-promtail"
        mountPath: "/etc/promtail.yaml"
        subPath:   "promtail.yaml"
      },
      {
        name:      "log"
        mountPath: "/var/log/modsec"
      },
    ]
  },
]
extraVolumeMounts: [
  {name: "log", mountPath: "/var/log/modsec"},
]
extraVolumes: [
  {name: "log", persistentVolumeClaim: claimName: "modsec-claim"},
  {name: "config-promtail", secret: {secretName:  "promtail-config"}},
]

Logrotate

On ajoute un container supplémentaire pour nettoyer les logs de ModSecurity.

La configuration ci-dessous provoque la rotation du fichier /var/log/modsec/audit.log (conservation 30j).

Puis une commande post rotate provoque la suppression des fichiers d'audit (on conserve 7 jours de logs)

{
    name:  "logrotate"
    image: "docker-registry.caascad.com/internal/logrotate:v1.4.1"
    env: [
        {name: "LOGS_DIRECTORIES", value:             "/var/log/modsec"},
        {name: "LOGROTATE_INTERVAL", value:           "hourly"},
        {name: "LOGROTATE_SIZE", value:               "100M"},
        {name: "LOG_FILE_ENDINGS", value:             "log"},
        {name: "LOGROTATE_MAXAGE", value:             "7"},
        {name: "LOGROTATE_POSTROTATE_COMMAND", value: "find /var/log/modsec/audit  -ctime +7 -delete"},
    ]
    volumeMounts: [
        {
            name:      "log"
            mountPath: "/var/log/modsec"
        },
    ]
},

Consultation des logs

Dans Grafana STG ou Grafana PRD, on peut rechercher les logs du job modsecaudit et filtrer sur l’application.

Exemple: {job="modsecaudit", app="ingress_controller_v2_bastion"} | json