Almost production

Pretty close to put the module in real use.

tasks

- make differnet name (at the moment kernel complains the module cannot be loaded as ng_netflow already there when original ng_netflow loaded) (UPD: Fixed, after figuring for 2 nights what's wrong just replaced all words "netflow" to new name "netfl2". Just changing self name wasn't enough, instead, it looks like symbol name of a proc or func was used by kernel to reference the name. Ok, this is a good reason to switch to another name right now, so the new name will be 'state').

- configure it process traffic as kurrent traffic shapers:
now they are implemented on linux:

-A FORWARD -p tcp -d XXXXXXXXX/19 --sport 1024:65535 -m length --length 500: -m hashlimit --hashlimit-mode dstip --hashlimit-above 15/sec --hashlimit-burst 80 --hashlimit-name P2P -j TOS --set-tos 2  -входящий к клиенту

in human words: All traffic, destined to XXXXX on higher ports and packet size 500 and higher exceeding 15 simultanious connections mark as tos2.
So, this introduces a kind of problem - there is no connection tracking info yet, so we don't know the connections no (yet).

The other problem, is that only certain  users should get this filter, but it's not desirable to put as many BPF programs, as with planned payload up to 6000 customers it isn't desirable to force a packet to go through 6000 BPF programs.

So, what can be done:

1.User-switch - an ng_user which holds ip table and switches traffic on two interfaces (match, notmatch) upon presence of the ip in the table.
User switch might also solve problem of transparent proxying (and user per-user services), as IP forwarding doesn't work on L2, but there is no L3 traffic processing in our architecture (DONE)


2.Do user-switching things in the netflow itself benefits is single search in flow cache will do the job, disadvantage is loss of flexibility and mixtrure of two modules, and ... yes, it against KISS.

thoughts about next steps

Spent one more day thinking about the most productive way of detecting the traffic and make whole thing realy fast, debuggable, managable, easy to use.

Idea to work with traffic in uni-directional way is definetly bad - most of the traffic signatures could be found only in one direction, but traffic could and should be identified in both.

So, the next major fix will be passing traffic through the netflow algo bi-directionally, and making policing for classes on both ways symmetrically.

This leads to major design changes - two bpf nodes needed, connecting to different classifying sides, flows ALWAYS should go through the netflow engine. This actually ok, as previously i mented to use the same setup on both traffic sides and now it is just taking alltogether in one begemoth.

One future thought - it's aren't that difficult to make a subscriber list and create a class 'prepend' for each subscriber - it means that each subscriber's traffic class could be handled separately. This is not a real need now, just thought for future, to include BRAS functionality

Ok, quite good.
Two classess, each class has it's own policy
To make it clear classes CIR differs 10 times. So screenshot is selfexplaining - i can only add that for test http traffic set as class 2 with CIR=100kbps and default class (all other traffic) CIR=10kbps. And... FreeBsd goes from FTP, while Ubunti from HTTP.

For convenience, i created a perl script with text config - to put tcpdump patterns with their descriptions and class numbers together. Later i'll consider reordering and on-fly changing on the same script.

What is next ?
- create real patterns - no ports, real DPI patterns, or patterns with ports.
- netlow2 node: new name and bi-directional functionality
- redefine names, terms, etc in terms upsteram and downstream.



I think that same should be done for classes - but, actually structure is pretty fixed there, so it's possible to create 50 (or 50k) classes and then only change classes CIR. Not so important now, just to consider in future

Starting with test setup on real traffic now.
I am altering testbed to make some ng_car tests, then will do ng_cars for default and clssified traffic, and then tests on classifiyng quality will start.

I am still not clear about general policy of classifying the traffic, most likely, it will be classification of "real-time" and defaulting rest to low-speed

One more thought: need to do significant netflow2 node rewriting - create upsteram and downstream links instead of data and in and think about linking opposite-direction flows (two times less work on classifying), and also think about possibility to use classes together with subscribers.