The initial research and lookup on modules and Go/Python possibilities.
First research and googling have revealed that it’s pretty standard features – to extend C functionality via using Go libs and there is a way to include and execute Python code inside C.
Then I finally decided to look deeper at existing Zabbix loadable module functionality.
I’ve already come across them before, while I was digging history syncers for some problems. That time I thought I might be better extending the History interface by making modules but then come to the conclusion that they have very very limited functionality to have a fully functional history interface to be implemented via modules.
Which is strange.
Yes, because it’s only a small step – to make a history read interface and then we would have a modular interface which would allow only by attaching a proper module to achieve all that fast and fancy-schmancy features like ClickHouse integration or, perhaps, any other History Storage which fits well for historical time series data.
So this has definitely to be done now. Fortunately, it’s quite simple functionality and I stress this again – I don’t understand why it hasn’t been done right out of the box.
Second. The load_modules procedure. It’s quite obvious and simple, just a few things I need to do there. I’ll need to do some extra functionality to do different processing of modules due to module callbacks that will be different.
Sure, new API_VERSION which I’ll call GLABER API VERSION must be declared by the module to be treated and loaded differently from original Zabbix modules.
In initial modules, there has been some intermediate structure to hold pointers to history callbacks responsible for different history types.
I don’t understand the profit of it. Maybe yet. But there is a big disadvantage to it.
I will be extending the list of callbacks: history, polling, perhaps preprocessing functions might be extended soon by custom community code.
So the intermediate structure would have to be changed each time when a new callback type appears.
This way all the modules will have to be rebuilt from sources on such a change. No good. So far I haven’t found a reason why wouldn’t use some predefined function names and use dlsym to find them and attach to a list of callbacks of certain functionality.
After digging all that and understanding Glaber code changes, I decided to start the initial implementation of the Go module.
But before doing something it’s always worth of searching – it’s very unlikely no one has done that already.
And that’s right: there is a nice place – Zabbix Share where there are plenty of modules and adaptors with examples how to code on Go, Python, PHP to make agent modules.
It’s not a big difference since it’s the same structure and functionality for agents. And some of them exist for ages. For example, the Golang module is dated by 2015 !.
This a little bit sad. I say this again. If Zabbix had had full history interface implemented, which only literally needs a few days from them, then today Zabbix would work on any storage thing for years already.
But ok, we can do it right now. It's never too late.
I have decided to start with Golang. It stays a bit apart from interpreted languages like Python/PHP. It’s compiled to a binary. It's faster.
Since some point after Go 1.3-1.5, it’s possible to build modules without tricks with renaming the “main” function which was mandatory before. Fully compiled to a binary code Go modules do not need adapters and might be directly built to shared libraries.
A few notes about Go itself – after looking through for it's features I have realized it’s just a perfect for most things I’ve already done in Glaber code: It has the very nice concept of channels that perfectly fits tasks like streaming flow of metrics to history storage. It has a lightweight pseudo-threading named "goroutines" which makes perfectly easy coding of async polling – you can code it in simple sync way, and Goroutines solve the problem of thread efficiency.
Sure, that all must be tested and checked first. I have some doubts if all that nice Go features will function in library mode when the only certain function will be called from C code. But even if they will not, it’s still worth using Go for code simplification.
Then, memory leaks. It is a concern now. Since there will be lots of new code running, it is something that must be checked well.
Especially in the case of Python. From my previous experience with a similar task with FreeRadius and rlm_perl it’s quite hard to eliminate memory leaks sometimes – too much code and functionality involved to be able to solve the thing in a reasonable time.
Ok, the actual tests. So far I have played a bit with compiling Go code to libraries and have tried to attach the libraries to a simple C code. Sometimes it has worked, sometimes it didn’t, but it’s enough to conclude that it’s perfectly doable and functional.
So the first plan will be to alter a bit Loading of Modules, implement support of history interface callbacks without touching existing modules support. And then after implementing basic reads and writes process a few millions of metrics, and look at my primary concerns about Golang:
1. Will go channels work?
2. Will we see any major memory leaks?
It’s most likely, that existing history_clickhouse will be moved to a separate module as well to compare C and Go efficiency if needed.
That’s it for now.
