Problems with BRAS high load ... fixed

About two weeks ago we've started to expereince problems with NAS servers is i wrote a week ago.

On peak load hours they were serving almost 300mbits and suffering from 6000+ pipes number. At some point dummynet caused degradation up to 10 times (both by traffic and packet no) Pipes where used for customer traffic policing.

To avoid the 'pipe' problem and get rid of dummynet i've wrote ng_dummynet module. The reason we don't want to use IPFW + tables + ng_ipfw - we are not letting L3 processing at all. Ng dummynet rememres which Ip in which class and sends traffic of the same class (which might be several IPs) to the same ng_car module.
Two days in production showed 20-30% degradation in comparison to dummynet which where very strange and unexpected. In two days i've spent lots of time trying to figure where is the problem, which could end up kerenl tasq process profiling.

But today on the tetst stand we finally detected that degradation happens on more than 1000 customers in new ng_dummynet module.

While doing block by block cut-offs of functionality in the module finally i've found the problem: it was diffent network and host byte order. The module uses hashing by two last bytes, which turned to be first two ip bytes and they are the same for all customers. So, the hashing woked as simple one by one IPs enumeration. Which turned to enumeration of 2-3k IPs in average for each packet.

The module fix worked out: imeddieately CPU tasq time dropped times.
Shure you want the picture:

The interesting thing: traffic from one NIC always processed on  one core.

Conclusions: on two NIC system it's dnagerous to work on 2-Core system as traffic starvation may overload all the system. On 4- kernel system (i5) with two NIC's CPU Utilization will not go higher then 50%, on 8-core i7 -  not higher then 25%. Of course on i5 and i7 free cores may be loaded with extra something, for example, put another NIC.

Take a look: С2Duo 8500 -> Core I7 (some higher model with 4 cores 3mhz) upgrade

And the last few words: i even started to write all-in one module to avoid multple IP lookups and rely only on netflow engine.
But now it seems to be useless. Unless i do P2P by behaviour recognition.

I think that after final optimizations we can get even more 10-15% CPU time on the same system.

ng_dummynet

Did some online debugging yesterday.

We've had one system with high load.

As for now i cannot exactly tell what is causing the high CPU load.

During the test i didn't touch the IPFW QUEUE system which is responsible for the policing. In fact, before reinitializing netgraph system experinced 70-80 CPU load (both cores the same 70-80) persent. At that moment there were 35-45K packets.

After netgraph reinit load dropped to 10-15 percent of each core. Pretty strange. The next thing to suspect was number of flows in ng_state. Before reinit there were about 120K. Right after init system collected 40K flows and load was 10-15 percent, in next 15 minutes it's rised to 80K flows, load raised substantioanaly - 2-5 percent.

Interestingly, complete removal of netgrpah processing doesn't result in significant CPU utulisation reduce - same 10-15% on tasq. Which means some part of netgraph is killing the performance.

So, to conclude: i am not sure that the problem is dummynet, but at least eliminating dummynet will allow to overcome 3K users (6k pipes) per system limit.

I've called it ng_dummynet. The thing is in process now. Yesterday i've did skeleton. Thing is supposed to be used together with ng_car which will be created per user basis. For now it is able to pass traffic uplink <-> downlink. Hoping to finish it by tommorow.

NAMED - almost end of story

03.04 NAMED - end of story
So, one week full-load flight with new recursor is finished.

Some results:
 - no software related problems except once we've expereinced some kind of attack when DNS traffic tripled, according to maintenance team report that was attack to the authoritattive servers, since they reside on the same hardware with recursors that caused significant system degradation, supposily, because of named was killing both CPU's, unfortunaly no real debugging and analyzing is possible now.
 - CPU load reaches 25-30% in peak time, and the good news powerdns as able to use both CPU cores without proccess blocking

Some setup details: actually now it is one cache, but with all logic put to two scripts - nxdomain and preresolve. Some auth functions related to different answers to different internal networks are put in preresolve scipt.

Local domains and RFC 1918 (grey) networks are forwarded to auth directly as root servers have no idea about their delegation (actually, they are site-specific zones), some black-listed zones are also processed in the preresolve. Scripts are in  c-perl-like LUA language, pretty simple and easy to underastnd language. According to tests even complicated lookups in LUA are much more fast end effective then doing real lookups (blacklisted zone).
Problems: they alwayas are. The only problem that most recent version of the pdns-recursor doesn't do round-robin DNS balancing correctly causing overloading of some servers. Previous version works fine, and thow, we left it in the production for now. The other thing, pdn-recursor also trims UDP and we cannot answer 40-50 server pools with it, BUT because of preresolve section we are'nt need it anymore as the problematic pool with ~50 servers is diveded to 6 networks and in LUA we can answer only those servers wich are supposed to serve that network segment, in comparison, BIND allowed only to do site-specific sorting of the pools, returning some servers first in the pool, but anyway all the pool was in the answer.


Overall: pdns-recursor is really nice upgrade, very low memory requerements, simple and efficient. Highly recomended for high recursor loads (5-20Kps of DNS traffic)

who is speaking

I am doing a litle out-of-job project - a callcenter for Itallians, to process paid services (mostly Future-Telling, Destiny, Erotic and so on). Actualy it's almost finished and now handles about 300-400 calls a day.

Some details - dedicated server receives calls via sip from PSTN via a  Carrier. Calls are routed to the operators. Some basic accounting, management is done in MySQL.

As the main developer dissaperaed in the middle, i am finishing the thing myself.

Actually this is pretty basic call-center, it's only interested me as i had no hands-in sip and voip systems experience before at all.

Now I've stopped development for a while and determine reason of frequent call drops in the evenings. To have a full picture i've installed ZABBIX localy and keep monitoring some basic aspects like connectivity to the sip provider and server load and other sites as well.