some news, good and not so good

At the moment we transferred almost 20k happy customers to new technology. 40k pending. The first outcome - about 20% of customers prefer old access methods.

Now we have problems which appeared as result of service simplification in the past. The problem is DUMMYNET. After 6k PIPES system dies. That's it, doesn't matter how much traffic.

So, now we are adding boxes to process traffic, but this is temporary solution. I look forward now to get rid of dummynet and use ng_car for policing.

When we;ve switched from dummynet on pptp servers to ng_car, we could double user capacity per system. Hopefully this will be the case this time again.

Some task analysis: there 2 primary methods to do the policing - one is to put separate traffic multiplexer to thousands of ng_cars, the other is to use well-working ng_state for that.

Advantage of separate multiplexer is it will just work in current setup. Disadvantage - every packet should be switched that way.
Implementation with ng_state allows to process only first packet in the flow, BUT following problems arrives: traffic forwarding and class policing should be done differently then. Actually instead of using different traffic path's for different classes i should add class labels to the packet.

Will do separate multiplexer now, as faster to implement solution and also, it might be not much slower comparing to the second solution as there label lookup will be needed for each packet.

DNS story, part2

Tests revealed that old djbdns seem to be too old as it processes requests 10 times slower then slow bind does.

The final decicsion and contest winner is powerDNS recursor for recursor part and BIND for authoritative part. And yes, finaly we've split them.

Idea to have 12 caches and split-horizon auth DNS server still continiuoes to be just an idea.

Instead it apperaed that powerDNS recursor has a very nice feature - it can pass all incoming requests to a lua script and all request ended with NX  answer to another lua script. This is enought to fullfill all our split horizon demands.

The only exception we don't know destination IP of DNS query in script when nice "packet cache" feature is on. Thats why at the moment two copies of powerDNS recursor is on duty. Not a big problem, two!=twelve.

It took about 10 working days to transfer all recursive payload from named to powerDNS. All went smooth,  without server lost.

Some things to notice:

• the most recent version of powerDNS recursor doesn't do round robin correct when answering queries with many resources. I didn't have time to figure out why, just downgraded to one minor version, it's ok.
• to have all private thing functional ( .local adresses and "grey" networks PTR resolving) recursor should be specifically told where auth server for them is, as root servers will not answer/know about such a resources. 
• restarts instantly, 
• memory consumtion as 200-300 megs, ten times less than named. 

At the moment we are leaving named as authoritative server. Mostly because of noc duty guys are comfortable to work with it, and absense of high traffic now, most mission-critical requests for customers processed inside recursor without asking auth server.

The CPU usage diagram:

Doesn't look impressive, but please note that user load (middle line) with bind couldn't get more then 50% (one CPU),  The user load magnitude after upgrade is because named cleaning it's empty cache (recursive part still on, but no real requests go to bind).

Bind, djbdns, etc and ISP

Today, instead of polishing my SCE solution to put it to production 4 am in Monday, i had to dig some DNS stuff.

In short, BIND sucks. To be more specific: actually it's not and it's a very good piece of software, which can do alot. But when used in so-called split-horizon setups it becomes very heavy, long starting, memory hungry begemoth

Today i spent 3 hours in trying to understand WTF is going on with one of our DNS servers. The task was even more complicated because of load balancers are on the way. I discovered it by very unpleasant surfing in browser, 'host' test showed that some queries, even cached ones took up to two seconds to resolve. That explained a lot.

So, when i looked at the first ns (say, ns1) stats - i saw this:

thats right, almost 100% resources under named. (See the green IDLE time).
Even at night time.

After doing some profiling ( strace -c ) i've discovered that most of the time named sits in futex syscall, and about 10-20% of them aren't succesifull.

Futex problem means concurrency problem. Switching off one CPU in config did magic thing - two out of three threads dropped futex time to less then 1%, and no more futex unsuccessful calls.

Interestingly, but the third process is almost only futex calls, and half of them failing.

This setup does the job much better - no visible delays occurs even when i put all the load to the server. And significant load drop.

Here is whole picture.

In the center there is the result of named restart - it's easy to see that problem not immediate, it takes 4-5 hours to occur.
On the right the result of switching to one CPU and getting rid of locks is visible - the system becomes 60% free, with no visible signs of degradation, actually,yet.

Troubleshooting is done, somehow it's working. But i need to further. First off all, hardware upgrade needed, as it's only c2duo 6300 CPU, will do on Monday.

But stop, any software can bring even most sophisticated hardware down to it's knees by doing stupid things like BIND does.

It looks like i had to look at something very efficient, for example djbdns, something like nginx.

The problem is that i need views support  (split-horizon DNS). Djbdns is the only free alternative able to do that. Others suggest setting extra instance of the server per view,  which will be pretty complicated setup in my case.

The other reason to get rid of BIND is for handling load-balanced pools which consists from 40-50 servers: BIND returns them all, and i haven't found a way to reduce number to say, 4-5 resources. And this part is really needed, because there are plenty of dumb devices that fail to resolve big RR pools.

Djbdns looks to be pretty efficient and realy nice solution.
It also splits caching and authoritive parts of the server into two daemons, which is nice industry-stnsart feature,but it is another problem for me.

Authoritive server will not do caching or forwarding, but it does split-horizon.

If i put caching server first (how it's supposed to be), then authoritive will have no info about user IPs which means that split horizon will not work.

Next i've got an idea to substitute root servers on authoritive server and use a patch to answer list of root servers when authoritive part recieves something it's not authoritive for. But it means every client will do two requests instead of one - first will return the IP of cache and then second will do the request to the cache. Not good.

What will definetly work - making one cache for each view. At the moment i have 5 views, and one view doesn't need caching.

So many thoughts that i am stopping at this point. Lets rest a bit.

While looking through the features of the servers i've found intreresting utulities with ProDNS server - they can replay captured DNS traffic on selected DNS - this is an easy way to simulate payload close to real on the new server.

Forwarding interesting traffic

We've spent a working day trying to figure how to implement forwarding of selected traffic in netgraph. No result. So ... It took a couple hours to do one one very simple module: fixmac.

Fixmac overwrites destination mac. This is forwarding, actually - one small disadvantage - need to handle MAC of destination router or system manually (unlike with IPFW, which works with arp table, but even IPFW never initiates ARP lookup).

Fixmac has two hooks (in and out) and process all traffic both ways. Everything going in -> out gets it mac replaced, out->in is intact.

Supports one netgraph message: ngctl msg fixmac: setmac 01:02:03:04:05:06, and stats.

Never thought i'll ever need to remember or get any practical use of this (src: wikipedia):

This time it was quite helpful.
most of my workmates failed to guess which goes first - ether addresses or 802.1q header.

ng_iptable load tests

Did high-load tests on the 2G test_bad - tol see the overall impact and simultaneous add/del/high_traffic to test stability

Now, switching back to state module:
First of all, need new testbed config and check if bi-directional flow recognition works. Simplified netgraph structure will be: 

So, i'am going to use same per-class netflow matching to do ip/based switching - which means iptable matching is done once per flow, if it would been on the main traffic path - then EACH packet had to be switched.
Most of important traffic will be send through 4 (if policing is needed) or 3 (no policing) nodes. "Learning" traffic may involve much more steps, in worst case packet has to be switched 50-100 bpf programs. This is CPU and delay problem, so, only first 6-8 packets in each flow are to be recognised.

The other thing, it might be desirable to add car to new flows, so automatically mark exceeding flows as being junk after certain limit (will do it later if needed).

One more thing (TODO) - "maintenance switch" - nobody likes when system crashes, and it does when reconfiguring nodes with high traffic on them. So, maintenance switch must be used to put traffic recognition OFF-line and so - do safe reconfiguration. 

ng_iptable

OOOps, tests shows that i did kernel panicer module, not iptable :)
    UPD: mutexes are to be inited prior to use :) - FIXED
    UPD: seem to be working, found bug with duplicate ip items - doing code to delete all duplicate entries -FIXED
Now, mass add/del - pecker test....
well, 10k add/removals... pretty stable ... yet no problems

ng_iptable

Decided to do user-IP-switch: it will be based on MUX module and will have three hooks (in, default, intable). A packet from "in" will be switched to default hook if it's not in the table and to "intable" if its in the ip table.
It will be able accept two additional control messages: ADD <IP> or DELETE <IP>.

Ok, renamed and cleaned the module, now it's capable of mixing traffic, but doesn't do ip lookup.
to do:
    - add ip , delete ip messages and addresses from the table. (DONE)
    - take ip from packet data (DONE, not checked yet)
    - do the search  (it has to be impressively effective) (DONE, but it's not even checked)

So, to check the thing will do the opposite setup of iptable and one2many nodes

This way: