
title: Ensure that Audit Log Errors Emit Alerts

description: |-
  <p>
  OpenShift audit works at the API server level, logging all requests coming to the server.
  However, if API server instance is unable to write errors, an alert must be issued
  in order for the organization to take a relevant action. e.g. shutting down that instance.
  </p>
  <p>
  Kubernetes by default has metrics that enable one to write such alerts:
  </p>

  <ul>
    <li><tt>apiserver_audit_event_total</tt></li>
    <li><tt>apiserver_audit_error_total</tt></li>
  </ul>

  Such an example is shipped in OCP 4.9+

  <pre>
  apiVersion: monitoring.coreos.com/v1
  kind: PrometheusRule
  metadata:
    name: audit-errors
    namespace: openshift-kube-apiserver
  spec:
    groups:
    - name: apiserver-audit
      rules:
      - alert: AuditLogError
        annotations:
          summary: |-
            An API Server instance was unable to write audit logs. This could be
            triggered by the node running out of space, or a malicious actor
            tampering with the audit logs.
          description: An API Server had an error writing to an audit log.
        expr: |
          sum by (apiserver,instance)(rate(apiserver_audit_error_total{apiserver=~".+-apiserver"}[5m])) / sum by (apiserver,instance) (rate(apiserver_audit_event_total{apiserver=~".+-apiserver"}[5m])) > 0
        for: 1m
        labels:
          severity: warning
  </pre>

  <p>
  For more information, consult the
  {{{ weblink(link="https://docs.openshift.com/container-platform/latest/logging/cluster-logging-external.html",
              text="official Kubernetes documentation") }}}.
  </p>

rationale: |-
  When there are errors writing audit logs, security events will not be logged
  by that specific API Server instance. Security Incident Response teams use
  these audit logs, amongst other artifacts, to determine the impact of
  security breaches or events. Without these logs, it becomes very difficult
  to assess a situation and do appropriate root cause analysis in such incidents.

references:
  nerc-cip: CIP-003-8 R5.1.1,CIP-003-8 R5.3,CIP-004-6 R2.3,CIP-007-3 R5.1,CIP-007-3 R5.1.1,CIP-007-3 R5.1.2
  nist: AU-5
  srg: SRG-APP-000109-CTR-000215

identifiers:
  cce@ocp4: CCE-90744-4

severity: high

ocil_clause: 'Audit log errors do not generate an alert'

ocil: |-
    Run the following command:
    <pre>{{{ ocil_oc_pipe_jq_filter('prometheusrules', '[.items[].spec.groups[].rules[].expr]', all_namespaces=True) }}} | grep apiserver_audit</pre>
    Make sure that there's a prometheus rule that verifies the <pre>apiserver_audit_error_total</pre> and 
    <pre>apiserver_audit_event_total</pre> metrics and will alert based on an error threshold.

warnings:
- general: |-
    {{{ openshift_filtered_cluster_setting({
        "/apis/monitoring.coreos.com/v1/prometheusrules": '[.items[].spec.groups[].rules[].expr]',
        }) | indent(4) }}}

template:
  name: yamlfile_value
  vars:
    ocp_data: "true"
    filepath: "{{{ openshift_filtered_path('/apis/monitoring.coreos.com/v1/prometheusrules', '[.items[].spec.groups[].rules[].expr]') }}}"
    yamlpath: "[:]"
    entity_check: "all"
    values:
    # Verify that the error ratio is calculated somehow
    - value: "^.*apiserver_audit_error_total.*apiserver_audit_event_total.*$"
      entity_check: "at least one"
      operation: "pattern match"
