NG_STATE
The ng_state module implements traffic policing with elements of DPI (deep packet inspection), classification and traffic processing according to classification results
What is it? A little bit of history, economics, reasons to use
Economic and common sense
The solution is resonable to ones who is dealing with > 1G of traffic, > 500 users to proccess and police, and does not have solition or needs functions the module has.
Most likely it's ISP applications, or corporate traffic gateways.
To make it up and running you need time and knowledge in FreeBSD administration and network systems integration.
I recommend to use Cisco devices or similar if possible. Due to their good support, stability and quality they are more likely to give higher service uptime results.
Nevertheless, if you are in situation of very fast growth, tight schedule, unable to buy or wait for such a device, have excessive amount of traffic, you are good in FreeBSD then the module and solution on it could be profitable.
The module might also be interesting if you already have a FreeBSD gateway.
Time
I estimate about 40 man-hours to start and test the solution, and 40 more to inetgrate it with external systems. One month to make it running in production systems.
Hardware
4-5Gigs of duplex traffix (4-5G up and down at the same time) is proccessed on 2k$ hardware,
10Gig in duplex – 3,2k$ – 4k$ for hardware.
History
At some point policing feature has te be split from existing NAS because of new type of access appeared. With such a split policing, proxying, messages, P2P detecion functions had to be done on standalone hardware.
Sure, we've spent a time looking for something ready out of box. The best thing to fit was Cisco SCE. We were ready to spent 30k$ per each processed Gig, and sacrifise some features,wait several month for delivery.
A device was ordered for test, but test is delayed for 2,5 month. Finally we've got it, but didn't like – the thing required another 3-4 PC to manage it (actually it turned out later that same 3-4 PC could do much more traffic processing then the device itself), couldn't alter traffic, only drop.
The device we've got for testing had one of it's two ports out of order so we've spent another 10 days in trying to make it work, chasing support, trying other Gbic/configs, chasing again and so on.
By the time we've finished the testing, all service was working on PC-based solution. It was working ok. It had some bugs, was not optimal, and CPU hungry.
Other few months of optimization, joining/cutting functions, centralizing processing turned to very effective, well-functioning module as we have now.
Appliance
The module implements user bandwidth policing according to service agreement and traffic type. It is intended to be used on Internet gateways.
Main features
- User bandwidth policing
- Possibility of having common CIR for group of IP addresses
- Traffic classification and traffic policing according to class
- Netflow v5 export
- WEB traffic redirect to display informational messages
- Flood and attack protection (internal and external)
- P2P traffic detection by behavior
The module must be put on a traffic path. It can be done by setting up standalone server with two NIC and connecting it into Ethernet link split or by installing module on an existing server.
It is preferable to install module on standalone server when bandwidth exceeds 500Mbit or 100kps.
The module is functionally close to Cisco SCE.
Some reference numbers
Number of supported users - 65k.
The module is tested on loads about 5-7Gbits, which is 4 million flows handling.
in theory the maximum number of flows is 40-50 million, user number is only limited to 65k by common sense.
Performance
Performance is really high. Traffic is handled on kernel level, and it goes to module almost without any other kernel processing, which is very efficient.
Also following should be considered
- NIC type and features
- CPU type, number of processors
- traffic characteristics
- Number of rules in classification module
Performance reference
100kps processed on one core of Core 2 Duo 6800 leads to 55-60% of one core utilization.
Core I7-960 machine with Intel 10Gb NIC is capable of processing about 2Mps which is 10-12Gbit total or 5-7Gbit in duplex.
In case when classification, Netflow and WEB traffic redirect is not required, and bandwidth policing is the only task, then its possible and desirable to switch off flow engine (at the moment this can be done by fixing module) which will lead to about 5x performance increase.
In such a configuration 180kps load takes 15% of one core of Core 2 Duo 6800 CPU.
The mentioned results where seen on average user inetnet traffic (40-50% is HTTP, another 40% is torrent).
System requirements
- ОS FreeBSD 8x. (although it works in 7X, 8 is recommended for better network subsystem)
- 2-4Gigs of memory
- 64 bit OS (preferred)
Installation
You will need kernel sources and you have to be sure that running kernel is the same version as sources are. If you are not - install kernel sources and then compile and install new kernel.
Adjust some OS parameters
kernel memory
netgraph items ( sometimes they help to survive congestion time )
Compile the module:
- Copy sources to /usr/src/sys/netgraph/state
- Copy makefile to /usr/src/sys/moduels/netgraph/state
- go to /usr/src/sys/moduels/netgraph/state, do make install clean
- load module kld_load ng_state
- Create you netgraph graph or use my script
- Using netgraph messages start setting services according to your needs
- Profit
Typical schemes
1. Standalone machine with classification
2. Standalone machine without classification
3. Existing machine
4. Making reserve
To make a reserve use any managable switch with STP/RSTP functionality. Switch bypass link In parralel to the ng_state server with higher STP/RSTP cost. This way whenever L2 connectivity is broken on the primary link, where the server is, then traffic switches to the bypass link. On some switches loop detection should be switched off in case all connectivity is done on the same device.
Remark: All non-IP traffic is travels without restrictions between up1 and down1 hooks. Such a traffic is not a subject of policing, so all L2 non-IP protocols will be running without policing and loss possibility.
Attention
Promisc mode
When working on Ethernet link split mode then NICs should be switched to promisc mode to disable hardware mac filtration.
Do for all NICs participating in the split
ifconfig em0: promisc
Traffic is processed on kernel space
On the time of critical load the system may become unmanageable, even from local console.
IPFW
As traffic does not get into IP processing level, firewall features related to IP doesn't work. Only basic ALLOW/DENY rules will do. Man ipfw, please.
MAC autorsrc for ng_ether
To disable source mac substitution in traffic, disable the feature in ng_ether of both NICs of the split.
ngctl msg em0: setautosrc 0
Classification
Classification feature
The module can get information about traffic class from external systems or modules. To do classification it's assumed that traffic can by classified on first 10 packets in the flow. After successive pattern match packets in the flow are classified with the same class and no more packets are sent to classifier.
As an example ng_bpf can be used. Ng bpf is capable of doing following:
- tcpdump expression
- packet length, headers
- protocol, ports
- patterns with payload data match
Man bpf, man ng_bpf please.
After being classified traffic is sent back to the ng_state modules to complete learning process:
Classified traffic processing
The module sends packets in it's original direction on hook up/down with number matching traffic class. If packet is not classified yet or hook with class number not exists then traffic is send out on hook 1. (up1 or down1)
Further processing examples
Overall view of classification process
- send packets to different NIC (say, sent all WEB traffic to NIC where transparent proxy waits to catch it)
- put additional policer to class hook (ng_car)
- send P2P to cheap channel
P2P
Module is capable of detecting p2p traffic by its behavior. Traffic that has been classified is excluded from detection. Detection is based on connections number and traffic intensity. Each 5 second for each service decision is made - if the service (user) is P2Per or not. For those detected P2Per traffic marked as p2p-pissible is sent to class 2 next 5 seconds.
Flood protection
Module forbids to create more the 4000 flows for each service, except default service. This protects from floods, port scans and other connection hungry attacks.
Note: i've tested a few p2p clients and was not able to reach the limit without p2p client tuning. Typycally about 300-500 connects are exists for 4-5 resources being downloaded by p2p. Even by doing tuning I was able to get 3000 flows. For normal use this limit shouldn't be a problem
Informational messages
.. are to display informational messages, for example 'out of balance' or agreement to use the service. All web traffic is sent to predefined IP address (destination IP address is changed, DNAT actually)
Managing the module
Service setup
For grouping several IPs to same group with same polices 'service' is used. Service is a set of common parameters and restrictions, such as CIR (commited information rate), p2p detection, flag to show informational messages). One IP can be at only at one service at a time. When IP is added to a new service, then it is removed from old one.
Module is controlled via netgraph messages.
Service and IP address manage commands
ngctl msg ng_state: setservice { userid=\”\” ip=192.168.0.1 cir_up=200000 cir_down=200000 fwd=1 fwd_ip=192.168.0.5 mpolicer=2 }
Sets service of customer identified by .
Cir_down, cir_up - CIR in downstream and upstream directions.
ip=192.168.0.5 to assign this ip to the service.
If it is desired to assign more addresses to the same service then do it by calling setservice command with IP and usename only.
ngctl msg state: setservice { userid=”” ip=192.168.0.2 }
One command for each IP address.
fwd – switch on traffic redirection of port 80 to display informational messages. Set fwd=2 to enable forwardingm fwd=1 to disable, if not set – then old value is kept.
fwd_ip - IP address where to send the traffic. Warning - this cannot be used for transparent proxy
mpolicer – P2P traffic detection for the service. To set use mpolicer=2. When not set (mpolicer=1), all traffic belonging to the service is ubject of P2P by behavior detection.
Service and address request commnads
ngctl msg ng_state: getservice { userid=”” }ngctl msg ng_state: getservice { ip=192.168.0.1 }
Will show service metrics for user identified by , or if only IP is given - then service will be found by the IP address.
Statistics
ngctl msg state: servicestat { userid=”” }ngctl msg state: servicestat { ip=192.168.0.5 }
Show per-service statistics – packets, bytes transmitted/dropped, flow number.
Mass speed check
ngctl msg state: getspeeds N
To do massive CIR verification to synchronize billing data and module settings. Returns 70-100 CIRs at a time (whatever number will fit 4kb buffer)
Hooks description
upX, downX. X - number of class. Packets, classified as class X will be leaving the module by hook upX or donwX (according to original destination). If hook is not connected, then up1/down1 is used.
To turn on packet class learning do
ngctl msg ng_state: setifupclass {iface=18 class=14}
All packets entering up18 will be classified as class 14. Flow the packet belongs will also be marked as class 14. All further packets in the flow will be classified as class 14 without learning. This way classes are set after matching rules in bpf-classifier
newflows_up, newflows_down – hooks which are used to sent new packets to. When classification is not possible for the packet then it should enter module again with class set to 10 (“unable to classify”) which will lead to sending further packets in the flow to classification. If classification fails after 10s packet, then flow is marked class 2 (P2P - possible ).
export – for netflow v5 export, works same way as ng_netflow.
Special features, defaults
Default service
Packets not belonging to any services are assumed to belong to "default service". Default service parameters might be set via setservice command with userid parameter set to ”default”
ngctl msg state: setservice { userid=\”default\” cir_up=64000 cir_down=64000 }
Same to retrieve service parameters.
ngctl msg state: getservice { userid=\”default\” }
Default service is supposed to be used as default policy – for example, when CIR speeds are set to very small values, then no traffic will go until IP is assigned to a service.
Deleting IP addresses from services
Not possible. But you can move ip to default service back. Use setservice
ngctl msg state: setservice {userid=\”defaut\” ip=127.0.0.2 }
Netflow
Export is done same way as ng_netflow. As the state module does eport in dedicated thread each 3-5 seconds - netflow stream is somewhat bursty.
In addition, src_ad field is used to pass number of service the flow belongs. Billing can check this number to see if correct service is set.
Predefined classes
- 1 – default class, shouldn't be policed
- 2 – p2p-possible class
- 10 – unable to classify yet
Please, Notice
The module is very specific instrument as almost useless alone. To make it fully functional it should be setup with scripts which will translate business-logic to the module commands.
Thanks
Special thanks to Gleb Smirnoff, whose ng_netflow is used as a base for the module.
here
