I decided to do that due to one thread of Xeon5645 couldn't handle all the preprocessing job, and also this might be required for faster CPU's but for higher loads.
First, some thoughts i had ahead:
Since there was already host-to-queue persistence in queuing version, so i did thread-to-preprocmanager persistence the same way (procnum%4). So all items from each host will get to the same preprocessing manager.
Workers are split by preprocessing managers in the same way, by hashing thair procnum.
So, after fixing the worker log handling problem, preprocessors could handle much more requests.I saw stable value around 110k per 5 seconds per each manager, giving altogether 88k NVPS. I thought of making a 100k, but decided not to waste time on this old hardware, but test something more up to date later.
By the way, measuring prprocessor manager statistics is the way how i prefer to measure NVPS. I assume server collects a bit more from the network. It does requests to inaccessible hosts, which return no data and don't appear at preprocessing manager.
So thats it. Splitting preproc manager is essential if there is a need to achieve more then 50k NVPS stable on modern CPU's. This is what actually was limiting the test server with Xeon E3-1280.
First, some thoughts i had ahead:
- 1.there might be a problem if two dependent items go to different managers so they will be waiting one another (deadlock)
- 2. It must be some kind of workers distribution.
Since there was already host-to-queue persistence in queuing version, so i did thread-to-preprocmanager persistence the same way (procnum%4). So all items from each host will get to the same preprocessing manager.
Workers are split by preprocessing managers in the same way, by hashing thair procnum.
So, after fixing the worker log handling problem, preprocessors could handle much more requests.I saw stable value around 110k per 5 seconds per each manager, giving altogether 88k NVPS. I thought of making a 100k, but decided not to waste time on this old hardware, but test something more up to date later.
By the way, measuring prprocessor manager statistics is the way how i prefer to measure NVPS. I assume server collects a bit more from the network. It does requests to inaccessible hosts, which return no data and don't appear at preprocessing manager.
So thats it. Splitting preproc manager is essential if there is a need to achieve more then 50k NVPS stable on modern CPU's. This is what actually was limiting the test server with Xeon E3-1280.
No comments:
Post a Comment