The guide to analyzing Kubernetes runtime detection alerts using Amazon Athena

0

Introduction

Lightspin has created a public repository with common use cases for simulating unusual/malicious activity within the Kubernetes cluster. Malicious activities include container evasion attempts, reconnaissance actions, and cryptocurrency mining. All presented use cases are detected by the Lightspin Kubernetes Runtime Protection solution which triggers alerts with comprehensive information about suspicious activity. You can stream alerts to an S3 bucket by setting up an “AWS S3” integration in the Lightspin platform. In this article, we’ll walk you through the process – from setting up an S3 bucket integration, to running a live simulation in a Kubernetes cluster, to Loading alerts from the S3 bucket to an Amazon Athena table and querying the results.

Before you start

You must have a Kubernetes cluster and the kubectl command line tool must be configured to communicate with your cluster. This cluster must also be connected to the Lightspin platform, and the “Execution Protection” the option must be enabled. Additionally, you must connect an AWS account to the Lightspin platform and ensure that you have access to Amazon Athena and S3 in that account.

Disclaimer: Please note that use of this document will incur charges to your AWS account related to the use of the Amazon Athena service and AWS S3 storage.

Configure S3 bucket integration in the Lightspin platform

If the AWS account is not already set up with an AWS S3 integration, follow the steps below to create one. You can create a new S3 bucket or use an existing one for the target S3 bucket.

  1. Open the Lightspin platform.
  2. Go to Settings and open Alerts and Notifications.
  3. In the Workspaces section, activate AWSS3 the integration.

  4. For AWS S3 bucket name insert the name of the target S3 bucket. For AWS Account choose the AWS account where the S3 bucket is located.

    2-1

  5. Click it Test the S3 bucket connection button to make sure the connection is successful.
  6. Click on to safeguard.
  7. In the Alert channels section, choose Create an alert channel.
  8. For Last nameenter a channel name for the new channel.
  9. For TypeChoose Kubernetes execution.
  10. For AWS S3 Bucketchoose the name of the S3 bucket you configured above.
  11. For Minimum gravityChoose Medium.
  12. Ensure the Enable option is checked.
  13. Click on to safeguard.

Congratulations! Got an S3 bucket integration in the Lightspin platform.

Simulate malicious activities and trigger events in the Kubernetes cluster

Now, when we set the S3 bucket integration, Kubernetes runtime protection alerts are broadcast to the bucket’s lightspin/k8s_runtime_events folder. We will perform simulations from lightspin-k8s-attack-simulations repository to trigger these events in the Kubernetes cluster. In this repository you can find the instructions to install and use this tool. We suggest running simulations to create multiple events in the S3 bucket.

In the example below, you can see the cryptocurrency mining simulation running inside the cluster.

3-1

Events from triggered alerts are added to the S3 bucket shortly after execution.

Create the Athena table

Amazon Athena provides a convenient and fast way to query data from S3 using SQL queries. With Athena, you can query large datasets and get results in seconds, and only pay for the queries you run. In the next section, we’ll create a table in Athena that will hold S3 runtime events.

  1. Open Console Athena.
  2. If this is your first time visiting the Athena console in your current AWS Region, choose Explore the query editor to open the query editor. Otherwise, Athena opens in the query editor.
  3. Picking out Display settings to configure a query result location in Amazon S3.
  4. In the Settings tab, choose Manage.
  5. For Manage settingsenter under Location of query result the path to a bucket where the query results will be stored. The bucket must be stored in the current AWS Region (Suggestion: don’t choose the same S3 bucket where you log runtime events).
  6. Picking out to safeguard.
  7. Picking out Editor and run the following query to create the table. Replace the BUCKET-NAME placeholder with the name of your S3 bucket that contains runtime events.

    CREATE EXTERNAL TABLE IF NOT EXISTS runtime_events (
    `container.id` string,
    `evt.time` string,
    `k8s.ns.name` string,
    `k8s.pod.name` string,
    `proc.cmdline` string,
    `proc.pid` int,
    `cluster_id` string,
    `rule_name` string,
    `description` string,
    `severity` string,
    `related_cves` string
    )
    ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
    LOCATION 's3://BUCKET-NAME/lightspin/k8s_runtime_events/'

The image below shows the query in the Athena editor:

4

The columns of the runtime_events table are defined as follows:

  • container.id: ID of the container where the event occurred
  • event.hour: The time when the event occurred
  • k8s.ns.name: The pod namespace
  • k8s.pod.name: The module name
  • proc.cmdline: Executed command line triggered event
  • proc.pid: The ID of the process generating the event
  • cluster_id: The identifier of the cluster on the Lightspin platform
  • rule_name: The name of the rule that detected the event
  • the description: The description of the rule that detected the event
  • gravity: how serious is the event
  • related_cves: if there are CVE related to this event

Run the command below to get a few rows and see what the data looks like:

SELECT * FROM runtime_events limit 10;

The image below shows an example of data output in the runtime_events table:

5

Analyze runtime events using Athena queries

After creating the runtime_events table, we can query the data and look for high priority alerts or additional information. Let’s look at some examples of interesting queries.

Query 1: Identify the most triggered rules

SELECT rule_name, severity, COUNT(rule_name) as count_rules FROM runtime_events GROUP BY rule_name, severity ORDER BY count_rules;

Example output:

6-1

Query 2: Find high severity alert events that occurred in the cluster

SELECT rule_name, "proc.cmdline", "evt.time" FROM runtime_events WHERE severity='High' ORDER BY "evt.time" DESC limit 8;

Example output:

seven

Query 3: Search for a specific rule

SELECT * FROM runtime_events WHERE rule_name="Detect Outbound Connections To Common Miner Pool Ports";

Example output:

8

Conclusion

According to Cloud Native Computing Foundation respondents in a 2021 study, 96% of organizations are using or evaluating Kubernetes – a record since their surveys began in 2016. Kubernetes has quickly become one of the most widely used services for managing containerized workloads and services in organizations. As such, it is essential that organizations improve their ability to secure and protect their environments. In this article, we have presented how to run an active test that simulates malicious attacks/activities within the Kubernetes cluster. Next, we followed the steps for creating an S3 bucket integration to stream and store runtime alert events from the Lightspin platform. Finally, we used Amazon Athena to create a table that loads data from the S3 bucket and analyzed the results with SQL queries. As the use of Kubernetes across regions and organizations continues to grow, it is critical that organizations can implement best practices and approaches to provide advanced protection.

*** This is a syndicated blog from the Security Bloggers Network of Lightspin Blog written by Ori Abargil. Read the original post at: https://blog.lightspin.io/the-guide-to-analyzing-kubernetes-runtime-detection-alerts-using-amazon-athena

Share.

Comments are closed.