AKS Traffic Analytics with NSG Flow Logs

Photo by Denys Nevozhai on Unsplash

Network traffic analysis in context of Azure Kubernetes Services managed cluster could be a complex task as it could involve installation of third-party products on the cluster’s Node, manually capturing network traffic with or without the necessary filters and exporting the results to an external destination in order to be analyzed. NSG FlowLogs combined with Traffic Analytics will simplify the process and would provide a significant support in troubleshooting network connectivity. A default installation of AKS cluster will provide a Network Security Group automatically configured by the AKS Resource Provider with the default rules for Inbound and Outbound traffic and also with the future services exposed from within the cluster. It is created by default in Infrastructure resource group (by default prefixed with MC_ ) and have the naming convention of aks — nodepool-name— nsg.

Activating the Flow Logs on the respective NSG is a straightforward process. We’ll open the Network Security Group Resource -> Monitoring -> NSF flow logs panel which will bring us in the default page. As we currently do not have a resource created, it will show us an empty list.

We choose the Create button, this will show us the first configuration panel for the resource, from the + Sign will select the NSG that we will monitor and will provide an existing or new Storage Account that will hosts the records in a Container. Also the retention day value should be provided in the same configuration panel.

On the second configuration panel, we choose the Version 2 of Flow Logs and also will activate the Enable Traffic Analytics button. After optional step of Tagging we choose to create the resource. Once with the Traffic Analytics selection, we will provide also a Log Analytics Workspace where our flow logging data will be saved. In a matter of seconds, we can see the resource created in our NSG Flow Logs page:

Once the NSG Flow Logs is linked to Log Analytics Workspace, we will go to Network Watcher -> Logs -> Traffic Analytics section in Azure Portal.

On top of the Traffic Analytics Panel we choose our resources, including the Resource Group where our NSG is located and also the Time Interval for results. It will provide a Dashboard with multiple values as follows:

- Number of flows

- Deployed Region and Vnet

- Total number of NSG monitored

• Subnets

Also the traffic distribution is included in the Report with the following:

- Total traffic

- Malicious traffic

- Blocked

• Frequent conversation flows.

For a raw capture of traffic, we can get the json output from the configured Storage Account. It will automatically create a container as follows:

In a hierarchical form created form Subscription → Resource Group -> Microsoft Network -> Network Security Group -> NodePool name -> year -> month -> day -> hour will be provided a Json formatted file. For a simple visualization of the file we can use the jq utility on the Linux side:

cat nsgflow.json | jq

{
  "records": [
    {
      "time": "2023-01-06T08:00:32.7787936Z",
      "systemId": "ef2a9241-0613-4394-b7d6-510a755cdb40",
      "macAddress": "6045BDF32344",
      "category": "NetworkSecurityGroupFlowEvent",
      "resourceId": "/SUBSCRIPTIONS/AA1792C8-2080-4570-9E12-A13C30464C9F/RESOURCEGROUPS/MC_AKS_AKS_WESTEUROPE/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/AKS-AGENTPOOL-21381009-NSG",
      "operationName": "NetworkSecurityGroupFlowEvents",
      "properties": {
        "Version": 2,
        "flows": [
          {
            "rule": "DefaultRule_AllowInternetOutBound",
            "flows": [
              {
                "mac": "6045BDF32344",
                "flowTuples": [
                  "1672991981,10.224.0.4,20.50.236.111,38088,443,T,O,A,B,,,,",
                  "1672991990,10.224.0.4,34.205.13.154,33952,443,T,O,A,B,,,,",
                  "1672991990,10.224.0.4,44.205.64.79,44122,443,T,O,A,B,,,,",
                  "1672991998,10.224.0.4,20.76.167.111,37756,443,T,O,A,E,14,3247,31,34068",
                  "1672992001,10.224.0.4,44.205.64.79,44122,443,T,O,A,E,1,74,1,74",
                  "1672992001,10.224.0.4,34.205.13.154,33952,443,T,O,A,E,17,4430,18,7707",
                  "1672992001,10.224.0.4,13.107.237.67,36734,443,T,O,A,E,35,2817,177,139974",
                  "1672992009,10.224.0.4,13.69.106.88,55146,443,T,O,A,E,2,120,3,525",

The complete documentation for the format of the NSG Flow Logs can be found at the following Microsoft documentation.

Traffic Analytics logs are sent to Log Analytics Workspace through a custom logging table. You can access the respective session from the link provided in the dashboard as follows:

There are three provided custom tables with the following schemas:

• AzureNetworkAnalytics_CL

• AzureNetworkAnalyticsIPDetails_CL