A short story about using PF_RING as solution to high CPU processing during traffic capture.
Our customer contacted ITORO with the following problem:
Traffic amount is about 19-20 Gb\s which is saturating our CPU and there drops during the sniffing process.
Wanguard is not able to see the whole traffic which leads to some attacks go through and reached our hosting services.
Solution: PF_RING ZC
ITORO has tuned the traffic capture drivers, optimized interrupts and RX queues and there were no more traffic drops.
This was a quick solution before deploying PF_RING ZC, which requires more changes in the underlying server configuration and some downtime for the whole system.
Since our customer used Intel 10 GE cards for traffic capture, filtering, and management connections - everything relied on the ixgbe driver.
This posted a few configuration problems, as with every change in the driver - all interfaces were inaccessible. The server needed to be accessed with an offline console to perform a full installation.
|There is a general recommendation to use separate and different cards to avoid putting the whole system on one relying driver.
For management connections to the server, it's a good practice to put those IPs on another network card like 1 GE or a different manufacturer. So when making changes to the 10 GE cards with the ixgbe driver, we can safely remove and reload driver without any disruption to management interfaces.
After generation of PF_RING ZC licenses and configuring the required settings for the driver we have moved to set Sniffing and Management Interfaces,
which is needed by PF_RING to distinguish between using stock ixgbe driver for normal communication and using ZC engine for sniffing interfaces.
The sniffing interfaces will be in kernel bypass mode, thus allowing the system to save CPU time and resources. The only disadvantage is, that the card can be accessed by only one application with its full speed.
Using PF_RING ZC engine allowed interfaces to run at full wire speed. The capturing traffic could reach full 10 Gb\s speeds on both ports.
As for the CPU we have noticed over 70% of savings with a comparison to standard PF_RING, giving Wanguard additional processing power for the filtering engine.
ITORO would like to thank NTOP.org team (Alfredo Cardigliano and Luca Deri ) for providing excellent support with fine-tuning their driver to achieve maximal performance.