Falco Runtime Security: Deep Dive

This deep dive explains how Falco works under the hood, the eBPF driver architecture, rule engine internals, alert pipelines, and how to build a comprehensive runtime security strategy.

Why Runtime Security Matters

Security does not stop at deployment time. You can scan images, enforce admission policies, and lock down RBAC, but threats can still emerge at runtime:

• A developer execs into a pod and runs a shell (lateral movement risk)
• A compromised container reads sensitive files or writes to system directories
• An attacker exploits a zero-day vulnerability after your image passed scanning
• A misconfigured pod spawns a crypto miner or tries to contact a C2 server

Shift-left security (scan early, fail fast) is essential, but it is not enough. You need runtime detection to catch threats that slip through.

The Defense in Depth Model

Modern Kubernetes security requires multiple layers:

Layer	Tool	What It Does
Prevent	Kyverno, OPA	Block bad configs at admission time
Scan	Trivy, Grype	Detect vulnerabilities in images before deployment
Detect	Falco	Catch suspicious behavior at runtime
Respond	Falcosidekick, PagerDuty	Alert teams, trigger incident response

Falco sits in the detection layer. It does not prevent anything. It observes and alerts.

How eBPF Works

Falco uses eBPF (extended Berkeley Packet Filter) to monitor kernel events without loading a kernel module.

What is eBPF?

eBPF is a Linux kernel technology that lets you run sandboxed programs inside the kernel. Originally designed for network packet filtering, it now supports tracing, profiling, and security.

Key characteristics:

• Safe: eBPF programs are verified before loading to ensure they cannot crash the kernel.
• Efficient: Programs run in kernel space, avoiding context switches.
• Non-intrusive: No kernel module required, no reboot needed.

Falco’s eBPF Driver

Falco installs an eBPF program that hooks into syscall events. Every time a process in a container makes a syscall (like open, execve, connect), the eBPF program captures it and sends it to Falco in userspace.

The flow:

1. Container process calls open("/etc/shadow", O_RDONLY)
2. Linux kernel triggers syscall entry hook
3. Falco’s eBPF program captures the event (syscall type, file path, process ID, container ID, etc.)
4. Event is sent to Falco’s userspace process via a ring buffer
5. Falco rule engine evaluates the event against loaded rules
6. If a rule matches, Falco emits an alert

Modern eBPF vs kernel module:

Falco supports three driver types:

• modern_ebpf: Uses eBPF (recommended, no kernel module)
• ebpf: Legacy eBPF driver
• kmod: Kernel module (fallback if eBPF is unavailable)

In this demo, we used driver.kind=modern_ebpf for the best balance of performance and compatibility.

Falco Rule Engine

Falco rules are written in YAML and evaluated against every syscall event. Each rule has three parts: condition, output, and priority.

Rule Anatomy

Here is a simplified version of the “Terminal shell in container” rule:

- rule: Terminal shell in container
  desc: A shell was spawned in a container with an attached terminal
  condition: >
    spawned_process and
    container and
    proc.name in (bash, sh, zsh, ksh, ash) and
    proc.tty != 0
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
    command=%proc.cmdline pid=%proc.pid)
  priority: WARNING

Breaking it down:

• condition: Boolean expression that matches events. If true, the rule fires.
• output: Template string for the alert message. Uses field references like %user.name.
• priority: EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG.

Field References

Falco provides hundreds of fields you can use in conditions and outputs:

Category	Examples
Process	proc.name, proc.cmdline, proc.pid, proc.ppid
File	fd.name, fd.directory, fd.type
User	user.name, user.uid, user.loginuid
Container	container.id, container.name, container.image
Kubernetes	k8s.pod.name, k8s.ns.name, k8s.deployment.name
Network	fd.sip, fd.dip, fd.sport, fd.dport

Macros and Lists

Falco supports macros (reusable condition fragments) and lists (arrays of values):

- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

- list: shell_binaries
  items: [bash, sh, zsh, ksh, ash, fish]

- rule: Shell in container
  condition: spawned_process and container and proc.name in (shell_binaries)
  output: "Shell detected (command=%proc.cmdline)"
  priority: WARNING

This makes rules easier to read and maintain.

Common Rule Patterns

Detect file access:

condition: >
  evt.type = open and
  fd.name = /etc/shadow and
  container.id != host

Detect network connections:

condition: >
  evt.type = connect and
  fd.sip = 0.0.0.0 and
  container

Detect process execution:

condition: >
  evt.type = execve and
  proc.name in (nc, ncat, socat)

Detect writes to system directories:

condition: >
  evt.type = open and
  evt.arg.flags contains O_CREAT and
  fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)

CNCF Graduated Status

Falco is a CNCF Graduated project. This is the highest maturity level in the CNCF landscape.

What Graduated means:

• Production-ready, widely adopted
• Strong governance and contributor diversity
• Comprehensive documentation and testing
• Regular releases with stability guarantees

Other Graduated projects include Kubernetes, Prometheus, Envoy, and Helm. Falco earning this status signals it is the de facto standard for runtime security in Kubernetes.

Alert Pipeline

Falco emits alerts to multiple destinations:

1. Standard Output (Logs)

By default, Falco writes alerts to stdout. You see them via kubectl logs.

Pros: Simple, built-in, works everywhere.
Cons: Alerts disappear when pods restart. No integration with external systems.

2. Falcosidekick

Falcosidekick is a companion project that forwards Falco alerts to external systems:

• Chat: Slack, Microsoft Teams, Mattermost
• Incident Management: PagerDuty, Opsgenie
• SIEM: Splunk, Elasticsearch, Datadog
• Messaging: Kafka, NATS, AWS SNS
• Storage: S3, Google Cloud Storage
• Webhooks: Generic HTTP endpoints

Enabling Falcosidekick:

helm install falco falcosecurity/falco \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  --set falcosidekick.config.slack.webhookurl=<YOUR_SLACK_WEBHOOK>

Now every Falco alert is posted to Slack in real time.

3. Falco Exporter

Falco Exporter exposes alerts as Prometheus metrics. You can create Grafana dashboards and Alertmanager rules:

helm install falco-exporter falcosecurity/falco-exporter

Metrics include:

• falco_events_total{priority="warning"} (counter)
• falco_events_total{rule="Terminal shell in container"} (counter)

Falco vs Other Runtime Security Tools

Tool	Approach	Strengths
Falco	Syscall monitoring via eBPF	Detects behavior, kernel-level visibility, CNCF graduated
Tracee (Aqua)	eBPF-based runtime security	Similar to Falco, focuses on attack signatures
Sysdig Secure	Commercial platform (built on Falco)	Enterprise features, managed rules, incident response
Tetragon (Cilium)	eBPF-based security observability	Policy enforcement + detection, network-aware

Falco is the most widely adopted open-source option. If you need commercial support or advanced features, Sysdig Secure is the commercial offering built on Falco.

Custom Rules

You can write your own rules to detect application-specific threats.

Example: Detect Pod Accessing External API

Suppose your app should never call the Kubernetes API server. You can create a rule:

- rule: Unauthorized K8s API Access
  desc: Detect container accessing Kubernetes API
  condition: >
    evt.type = connect and
    fd.sip = 0.0.0.0 and
    fd.dip = 10.96.0.1 and
    fd.dport = 443 and
    container.image contains "my-app"
  output: >
    Unauthorized K8s API access
    (container=%container.name pod=%k8s.pod.name)
  priority: ERROR

Replace 10.96.0.1 with your cluster’s API server ClusterIP (kubectl get svc kubernetes -n default).

Example: Detect Specific File Access

Detect when a container reads your application’s secret config:

- rule: Config File Read
  desc: Detect reads to application config
  condition: >
    evt.type = open and
    fd.name = /app/config/secrets.json and
    container.name != admin-pod
  output: >
    Secrets file read by unauthorized container
    (container=%container.name user=%user.name)
  priority: CRITICAL

Loading Custom Rules

Option 1: ConfigMap

Create a ConfigMap with your custom rules and mount it into the Falco pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: falco-custom-rules
  namespace: falco-system
data:
  custom-rules.yaml: |
    - rule: My Custom Rule
      condition: ...
      output: ...
      priority: WARNING

Update Falco Helm values to load it:

customRules:
  custom-rules.yaml: |-
    - rule: My Custom Rule
      ...

Option 2: Helm Values

Pass custom rules directly in values.yaml:

customRules:
  my-rules.yaml: |-
    - rule: Detect Package Manager
      condition: >
        spawned_process and
        proc.name in (apk, apt, yum, dnf)
      output: "Package manager run in container (command=%proc.cmdline)"
      priority: WARNING

Reinstall Falco with the updated values.

Tuning False Positives

Falco can be noisy in some environments. Here is how to reduce false positives:

1. Exclude Known-Good Behavior

If your CI/CD pipeline regularly execs into pods, exclude those pods:

- rule: Terminal shell in container
  condition: >
    spawned_process and
    proc.name in (bash, sh) and
    not k8s.ns.name in (ci-system, ci-runners)
  output: ...
  priority: WARNING

2. Use append to Extend Rules

Do not edit the default rules directly. Use append in custom rules:

- rule: Terminal shell in container
  append: true
  condition: and not k8s.pod.label.ignore-falco = "true"

This adds the condition to the existing rule without overwriting it.

3. Adjust Priorities

If a rule fires too often and is not actionable, downgrade it from WARNING to INFO:

- rule: Terminal shell in container
  override:
    priority: INFO

4. Disable Rules

Disable a rule entirely:

- rule: Write below binary dir
  enabled: false

Only do this if you are sure the rule is not useful in your environment.

Building a Runtime Security Strategy

Falco is a detection tool, not a prevention or response tool. To build a complete strategy:

1. Define Baseline Behavior

Run Falco in audit mode for a week. Collect all events. Identify normal patterns (CI jobs, health checks, monitoring agents). Write rules to exclude known-good behavior.

2. Set Alert Thresholds

Not every alert requires an immediate response. Use priority levels:

• CRITICAL/ERROR: Page the on-call engineer immediately.
• WARNING: Post to Slack, review during business hours.
• INFO: Log for forensic analysis, no active monitoring.

3. Integrate with Incident Response

Connect Falco to your incident response workflow:

• Alerts -> PagerDuty: Trigger incidents for CRITICAL alerts
• Alerts -> SIEM: Correlate with other security events
• Alerts -> Webhooks: Trigger automated remediation (kill pod, isolate network)

4. Combine with Other Tools

Runtime security is one layer. Combine with:

• Admission control (Kyverno): Prevent bad configs
• Image scanning (Trivy): Block vulnerable images
• Network policies: Restrict pod-to-pod traffic
• RBAC: Limit who can exec into pods

5. Regular Review

Review Falco rules quarterly. Remove rules that never fire. Add new rules as threats evolve.

Performance Considerations

Falco has low overhead, but high-traffic systems may need tuning.

CPU and Memory

Falco typically uses:

• CPU: 0.1-0.5 cores per node
• Memory: 100-300 MB per node

High-throughput systems (1000+ syscalls/sec) may see higher usage.

Buffer Sizing

Falco uses a ring buffer to store events before processing. If the buffer fills up, events are dropped.

Tune buffer size via Helm:

driver:
  ebpf:
    bufSizePreset: 4

Presets: 1 (smallest) to 8 (largest). Larger buffers reduce dropped events but use more memory.

Dropped Events

Check for dropped events:

kubectl logs -l app.kubernetes.io/name=falco -n falco-system | grep "Falco internal: syscall event drop"

If you see drops, increase buffer size or reduce rule complexity.

Falco in Production

Best practices for running Falco in production clusters:

1. DaemonSet with Tolerations

Falco should run on every node, including control plane nodes:

tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane

2. Resource Limits

Set resource limits to prevent Falco from consuming too much:

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

3. Persistent Storage

Use persistent storage for rule files and configuration:

volumes:
  - name: rules
    persistentVolumeClaim:
      claimName: falco-rules

4. Multi-Cluster Monitoring

For large organizations with many clusters, centralize alerts:

• Run Falco in each cluster
• Forward alerts to a central SIEM (Splunk, Elasticsearch)
• Create cross-cluster dashboards in Grafana

5. High Availability

Falco is stateless, so there is no HA configuration needed. Each node runs its own Falco pod. If a pod crashes, kubelet restarts it. Events are not lost (they are generated from kernel hooks, not stored in Falco).

Comparing Detection Approaches

Runtime security tools use different detection methods:

Approach	How It Works	Pros	Cons
Syscall Monitoring	Hook into kernel syscalls (Falco, Tracee)	Deep visibility, detects zero-days	High volume of events, needs tuning
Behavioral Analysis	Machine learning on process behavior (Datadog, Sysdig)	Low false positives	Requires baseline, may miss novel attacks
File Integrity Monitoring	Detect file changes (AIDE, Tripwire)	Simple, low overhead	Reactive, does not detect memory attacks
Network Anomaly Detection	Analyze traffic patterns (Cilium Hubble)	Detects C2 traffic	Misses local attacks, encrypted traffic is opaque

Falco uses syscall monitoring. It is the most comprehensive approach but requires tuning to reduce noise.

Extending Falco

Falco is highly extensible:

1. Plugins

Falco supports plugins for custom data sources:

• k8saudit: Parse Kubernetes audit logs
• cloudtrail: Detect AWS API threats
• okta: Monitor identity events

Install a plugin via Helm:

plugins:
  - name: k8saudit
    library_path: /usr/share/falco/plugins/libk8saudit.so

2. Custom Outputs

Write your own output handler in Go or Python. Connect Falco to proprietary systems.

3. gRPC API

Falco exposes a gRPC API for programmatic access:

helm install falco falcosecurity/falco \
  --set grpc.enabled=true

Use the API to build custom dashboards, incident response tools, or compliance reports.

Summary

Falco is the industry standard for Kubernetes runtime security. It uses eBPF to monitor syscalls, matches events against rules, and emits alerts when threats are detected.

Key takeaways:

• Runtime security is essential (shift-left is not enough)
• eBPF provides deep kernel visibility without kernel modules
• Falco rules are flexible and powerful (hundreds of built-in rules, easy to customize)
• Integrate with Falcosidekick, Prometheus, and SIEM for automated response
• Combine with admission control (Kyverno), scanning (Trivy), and network policies for defense in depth

Falco is CNCF Graduated, production-ready, and widely adopted. If you are serious about Kubernetes security, Falco should be part of your stack.

Further Reading

• Falco Documentation
• CNCF Security Whitepaper
• eBPF Introduction
• Falcosidekick Outputs
• Sysdig Secure (commercial Falco-based platform)