Cisco storm control and VMware Virtual SAN 6 Multicast Throughput problems

While deploying Virtual SAN 6 on Cisco hosts for a customer in a existing environment we got snagged by a few buggers.

First we had the issue that on the Cisco hosts, Virtual SAN was working but we kept some communication errors.
Eventually this was a firewall issue between VLANS. The new Virtual SAN cluster was build from scratch but it’s vCenter was placed on a existing management platform. During deployment not all required Virtual SAN ports were opened by the networking guys between the VLANS causing strange Virtual SAN behaviour.

Second problem was that the Multicast throughput was a between 0.00 and 17 MB/s. Not good 🙂
We first thought that this was also blocked by the firewall but that was not the case.

The default multicast group ports for Virtual SAN are:
Port: 12345
Port: 23451

Troubleshooted the life out of this one..
The already in place Cisco Nexus switches had the correct configurations for Multicast, IGMP Snooping and a IGMP Querier.

Eventually the culprit was the Traffic Storm Control feature of Cisco.
This caused to drop almost all Virtual SAN network packages..
Disabling this feature on the specific ports solved the Virtual SAN Multicast throughput issue!
Fun thing is that this feature is not on by default and was put on by someone..
Somehow with IGMP Snooping and a IGMP Querier, Traffic Storm Control still thought this was a flood of packages to be killed..

Some reading material on those topics below.

Multicast with IGMP Snooping and an IGMP Querier

IGMP Snooping

Layer 2 multicast forwarding, without IGMP snooping and an IGMP Querier enabled, is essentially a layer 2 network broadcast. Each network device attached to an active network port will receive the multicast network traffic.

IGMP Snooping and an IGMP Querier can be leveraged to constrain the IPv4 multicast traffic to only those switch ports that have devices attached that request it. This will avoid causing unnecessary load on other network devices in the layer 2 segment by requiring them to process packets that they have not solicited (similar to a denial-of-service attack).

Understanding Traffic Storm Control

A traffic storm occurs when packets flood the LAN, creating excessive traffic and degrading network performance. The traffic storm control feature prevents LAN ports from being disrupted by a broadcast, multicast, or unicast traffic storm on physical interfaces.

Traffic storm control (also called traffic suppression) monitors incoming traffic levels over a 1-second traffic storm control interval, and during the interval it compares the traffic level with the traffic storm control level that you configure. The traffic storm control level is a percentage of the total available bandwidth of the port. Each port has a single traffic storm control level that is used for all types of traffic (broadcast, multicast, and unicast).

Traffic storm control monitors the level of each traffic type for which you enable traffic storm control in 1-second traffic storm control intervals.

In all releases, and by default in Release 12.2(33)SXJ and later releases, within an interval, when the ingress traffic for which traffic storm control is enabled reaches the traffic storm control level that is configured on the port, traffic storm control drops the traffic until the traffic storm control interval ends. Release 12.2(33)SXJ and later releases support these configurable traffic storm control optional actions:

Shutdown—When a traffic storm occurs, traffic storm control puts the port into the error-disabled state. To reenable ports, use the error-disable detection and recovery feature or the shutdown and no shutdown commands.
Trap—When a traffic storm occurs, traffic storm control generates an SNMP trap.

Figure 57-1 shows the broadcast traffic patterns on a LAN interface over a specific interval. In this example, traffic storm control occurs between times T1 and T2 and between T4 and T5. During those intervals, the amount of broadcast traffic exceeded the configured threshold.

Figure 57-1 Broadcast Suppression

The traffic storm control threshold numbers and the time interval combination make the traffic storm control algorithm work with different levels of granularity. A higher threshold allows more packets to pass through.

Traffic storm control is implemented in hardware. The traffic storm control circuitry monitors packets passing from a LAN interface to the switching bus. Using the Individual/Group bit in the packet destination address, the traffic storm control circuitry determines if the packet is unicast or broadcast, keeps track of the current count of packets within the 1-second interval and when the threshold is reached, traffic storm control filters out subsequent packets.

Because hardware traffic storm control uses a bandwidth-based method to measure traffic, the most significant implementation factor is setting the percentage of total available bandwidth that can be used by controlled traffic. Because packets do not arrive at uniform intervals, the 1-second interval during which controlled traffic activity is measured can affect the behavior of traffic storm control.

The following are examples of traffic storm control behavior:

If you enable broadcast traffic storm control, and broadcast traffic exceeds the level within a 1-second traffic storm control interval, traffic storm control drops all broadcast traffic until the end of the traffic storm control interval.
If you enable broadcast and multicast traffic storm control, and the combined broadcast and multicast traffic exceeds the level within a 1-second traffic storm control interval, traffic storm control drops all broadcast and multicast traffic until the end of the traffic storm control interval.
If you enable broadcast and multicast traffic storm control, and broadcast traffic exceeds the level within a 1-second traffic storm control interval, traffic storm control drops all broadcast and multicast traffic until the end of the traffic storm control interval.
If you enable broadcast and multicast traffic storm control, and multicast traffic exceeds the level within a 1-second traffic storm control interval, traffic storm control drops all broadcast and multicast traffic until the end of the traffic storm control interval.