ng_state

Working on releasing the ng_state code and description.
Created google code project.
Documentation is almost ready, polishing now.

Engrish

I've just read all my posts. My English is ugly. I am sorry about that. I will try to make it better. Thanks.

DPI traffic recognition

How it works
To survive congestion times and prioritize traffic we're using mix of DPI and behavior-analyze.

DPI looks signatures and marks flows as certain class if they match patterns.

If flow doesn't match a pattern after first 10 packets it is marked as P2P-possible.
Each 5 seconds the decision is made about each client - is he P2P-er or not.

For ones detected as P2P-eer P2P-possible traffic is subject of additional polices.

Why behavior ?
 - P2P traffic mutates frequently and developers do everything to hide it, so DPI is difficult to maintain and not effective
 - such a traffic easily detectable by it's behavior - P2P applications tend to create hundreds of `connections` (TCP or UDP), and they greedy about bandwidth

So, to detect P2P it's simply enough to count flows and amount data transferred via that flows.
We also do simple optimizations - only considering  P2P-possible flows where traffic have been seen in last 3 seconds.

The numbers  were picked after 2 days of capturing and analyzing flow statistics.

One of the methods to do that was creating graphs from real-life.
This is how some kinds of traffic looks like:

Web pages generate up to 100 flows, but they contain relatively small number of bytes.

Downloads from one host (say, ftp or http file download) give ralatively small number of flows with big amount of bytes in them.

P2P traffic generates either a few hundreds number of flows  with lots of bytes in active flows.

P2P traffic is very flexible - according to network conditions it can do big flows from single peer or download only a few hundend kilobytes from each, but do it from thousand of peers. We've tried to do per-flow policing while detecting P2P - this gives no result - P2P software adapts very well to such a conditions: if large flows restricted - does lots of small flows. 

Policing traffic by it's behavior is a "feedback" system.
So, policing parameters should be adequate to now allow the "feedback" system to start flapping between border states (RED), instead it should attenuate to stable level.

So, playing with timings and amount of regulation each time we've got the BLUE picture bellow.

By now the system is pretty effective.
It's possible that later P2P will mutate to look like one of white-list protocols, this is very simple to do, but i think it won't be difficult to detect that.

At the moment from real traffic we weren't able to find such things.
But we've discovered that it's rather popular to use HTTP in games, SSL in applications, but haven't detected any P2P look-like traffic on top of them.

And the last, economical part:

We'd like to buy a ready DPI solution, because it has so many patterns, they are updated, maintained, BUT all of them COSTS.

Prices are about 10k$ per Gig. Our BRAS and DPI and SCE combined-device on top of PC hardware is 0.5k$ per Gig, and yes, plus our time, but still more then 10 times difference, plus easy upgradable and expandable.

Problems with BRAS high load ... fixed

About two weeks ago we've started to expereince problems with NAS servers is i wrote a week ago.

On peak load hours they were serving almost 300mbits and suffering from 6000+ pipes number. At some point dummynet caused degradation up to 10 times (both by traffic and packet no) Pipes where used for customer traffic policing.

To avoid the 'pipe' problem and get rid of dummynet i've wrote ng_dummynet module. The reason we don't want to use IPFW + tables + ng_ipfw - we are not letting L3 processing at all. Ng dummynet rememres which Ip in which class and sends traffic of the same class (which might be several IPs) to the same ng_car module.
Two days in production showed 20-30% degradation in comparison to dummynet which where very strange and unexpected. In two days i've spent lots of time trying to figure where is the problem, which could end up kerenl tasq process profiling.

But today on the tetst stand we finally detected that degradation happens on more than 1000 customers in new ng_dummynet module.

While doing block by block cut-offs of functionality in the module finally i've found the problem: it was diffent network and host byte order. The module uses hashing by two last bytes, which turned to be first two ip bytes and they are the same for all customers. So, the hashing woked as simple one by one IPs enumeration. Which turned to enumeration of 2-3k IPs in average for each packet.

The module fix worked out: imeddieately CPU tasq time dropped times.
Shure you want the picture:

The interesting thing: traffic from one NIC always processed on  one core.

Conclusions: on two NIC system it's dnagerous to work on 2-Core system as traffic starvation may overload all the system. On 4- kernel system (i5) with two NIC's CPU Utilization will not go higher then 50%, on 8-core i7 -  not higher then 25%. Of course on i5 and i7 free cores may be loaded with extra something, for example, put another NIC.

Take a look: С2Duo 8500 -> Core I7 (some higher model with 4 cores 3mhz) upgrade

And the last few words: i even started to write all-in one module to avoid multple IP lookups and rely only on netflow engine.
But now it seems to be useless. Unless i do P2P by behaviour recognition.

I think that after final optimizations we can get even more 10-15% CPU time on the same system.

ng_dummynet

Did some online debugging yesterday.

We've had one system with high load.

As for now i cannot exactly tell what is causing the high CPU load.

During the test i didn't touch the IPFW QUEUE system which is responsible for the policing. In fact, before reinitializing netgraph system experinced 70-80 CPU load (both cores the same 70-80) persent. At that moment there were 35-45K packets.

After netgraph reinit load dropped to 10-15 percent of each core. Pretty strange. The next thing to suspect was number of flows in ng_state. Before reinit there were about 120K. Right after init system collected 40K flows and load was 10-15 percent, in next 15 minutes it's rised to 80K flows, load raised substantioanaly - 2-5 percent.

Interestingly, complete removal of netgrpah processing doesn't result in significant CPU utulisation reduce - same 10-15% on tasq. Which means some part of netgraph is killing the performance.

So, to conclude: i am not sure that the problem is dummynet, but at least eliminating dummynet will allow to overcome 3K users (6k pipes) per system limit.

I've called it ng_dummynet. The thing is in process now. Yesterday i've did skeleton. Thing is supposed to be used together with ng_car which will be created per user basis. For now it is able to pass traffic uplink <-> downlink. Hoping to finish it by tommorow.

NAMED - almost end of story

03.04 NAMED - end of story
So, one week full-load flight with new recursor is finished.

Some results:
 - no software related problems except once we've expereinced some kind of attack when DNS traffic tripled, according to maintenance team report that was attack to the authoritattive servers, since they reside on the same hardware with recursors that caused significant system degradation, supposily, because of named was killing both CPU's, unfortunaly no real debugging and analyzing is possible now.
 - CPU load reaches 25-30% in peak time, and the good news powerdns as able to use both CPU cores without proccess blocking

Some setup details: actually now it is one cache, but with all logic put to two scripts - nxdomain and preresolve. Some auth functions related to different answers to different internal networks are put in preresolve scipt.

Local domains and RFC 1918 (grey) networks are forwarded to auth directly as root servers have no idea about their delegation (actually, they are site-specific zones), some black-listed zones are also processed in the preresolve. Scripts are in  c-perl-like LUA language, pretty simple and easy to underastnd language. According to tests even complicated lookups in LUA are much more fast end effective then doing real lookups (blacklisted zone).
Problems: they alwayas are. The only problem that most recent version of the pdns-recursor doesn't do round-robin DNS balancing correctly causing overloading of some servers. Previous version works fine, and thow, we left it in the production for now. The other thing, pdn-recursor also trims UDP and we cannot answer 40-50 server pools with it, BUT because of preresolve section we are'nt need it anymore as the problematic pool with ~50 servers is diveded to 6 networks and in LUA we can answer only those servers wich are supposed to serve that network segment, in comparison, BIND allowed only to do site-specific sorting of the pools, returning some servers first in the pool, but anyway all the pool was in the answer.


Overall: pdns-recursor is really nice upgrade, very low memory requerements, simple and efficient. Highly recomended for high recursor loads (5-20Kps of DNS traffic)

who is speaking

I am doing a litle out-of-job project - a callcenter for Itallians, to process paid services (mostly Future-Telling, Destiny, Erotic and so on). Actualy it's almost finished and now handles about 300-400 calls a day.

Some details - dedicated server receives calls via sip from PSTN via a  Carrier. Calls are routed to the operators. Some basic accounting, management is done in MySQL.

As the main developer dissaperaed in the middle, i am finishing the thing myself.

Actually this is pretty basic call-center, it's only interested me as i had no hands-in sip and voip systems experience before at all.

Now I've stopped development for a while and determine reason of frequent call drops in the evenings. To have a full picture i've installed ZABBIX localy and keep monitoring some basic aspects like connectivity to the sip provider and server load and other sites as well.