specter, userspace logging daemon.
specter makes use of the Linux >= 2.4.x packet filter subsystem (iptables) and
the ULOG target for iptables.
I want to provide a flexible, almost universal logging daemon for netfilter ULOG target. It is not optimized in any way, the goal is to keep as simple as possible. These are my thoughts about how the architecture which is most capable of doing that:
It should be possible to add plugins / runtime modules for new protocols, etc. For example the standard logging daemon provides source-ip, dest-ip, source-port, dest-port, etc. Logging for variuos other protocols (GRE, IPsec, ...) may be implemented as modules.
... describe how and where to put the information gained by input plugins. The easiest way is to build a line per packet and fprint it to a file. Some people might want to log into a SQL database or want an output conforming to the intrusion detection systems communication draft from the IETF.
The major clue is providing a framework which is as flexible as possible. Nobody knows what strange network protocols are out there :) Flexibility depends on the communication between the output of the input plugins and input of the output plugins.
Harald, following Rusty's advise, implemented type-key-value triples,
which work quite well for that purpose. Structure used for exchanging
data between input and output plugins is defined in specter.h, and
is called specter_iret_t. Most of time, output plugins precisely know what
data they need, so there must exist good querying system, as input
keys are dynamically defined and stored. Up to ulogd 0.3 this was done
by several linked list iterations, which weren't obviously very fast.
In 0.9 Harald implemented usage of hash tables initialized during
init. The idea was good, but deep levels of data structures one
had to dig into to get simple value and somewhat obscure style
(like accessing ulogd_keyh[] from the inside of plugin) forced
me (Michal) to rewrite this again. That's when fork from ulogd happened.
Abandoning hash tables, specter implementation use only pointers
accessed by general function find_iret(). To simplify usage of
that pointers, simple data structure specter_local_ret_t and
few macros defined in plugins/lret.h were also created.
Note they're not the obligatory extension; one can create his own
implementation based on single find_iret() definition.
First you will need a 2.4.x or 2.6.x kernel. If you have a kernel >= 2.4.18-pre8, it already has the kernel suport for ULOG (ipt_ULOG.o).
If you have an older kernel version (between 2.4.0 and 2.4.18-pre6), you can use the patch-o-matic system of netfilter/iptables, as described in the following section.
If you experience problems like described in my mail http://lists.netfilter.org/pipermail/netfilter-devel/2004-June/015860.html you should apply ipt_ULOG patch you can find in contrib/ subdirectory. For some reason it wasn't merged into netfilter source.
You only need to read this chapter if you have a 2.4.x kernel <= 2.4.18-pre6.
In order to put the ipt_ULOG module into your kernel source, you need the latest iptables package, or even better: the latest CVS snapshot. A description how to obtain this is provided on the netfilter homepage http://www.netfilter.org/.
To run patch-o-matic, just type
make patch-o-matic
Download the specter package from http://joker.linuxstuff.pl and untar it.
If you want to build specter with MySQL support, type './configure --with-mysql'. You may also have to specify the path of the mysql libraries using '--with-mysql=path'. To build specter without MySQL support, just use './configure'.
The same procedure apply to PostgreSQL support (use './configure --with-pgsql' with or without path to libraries).
If you have other applications using libipulog library contained with this package, you may consider building it shared. To enable this, use './configure --with-sharedlib'.
To compile and install the program, call 'make install'.
There are several front-ends for viewing logs generated by specter. Although they where designed for ulogd, there should be no problem with using them with specter. Here are few links to that kind of projects:
You can also find sample php query script in contrib/.
Just add rules using the ULOG target to your firewalling chain. A very basic example:
iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --ulog-prefix foo
To increase logging performance, try to use the
--ulog-qthreshold N
Of course you can combine the ULOG target with the different netfilter match modules. For a more detailed description, have a look at the netfilter HOWTO's, available on the netfilter homepage.
The number of the netlink multicast group to which ULOG'ed packets are sent. In specter, you can specify different task for different netlink groups, see specter configfile syntax reference section for more details.
Copyrange. This works like the 'snaplen' paramter of tcpdump. You can specify a number of bytes up to which the packet is copied. If you say '40', you will receive the first fourty bytes of every packet. Leave it to '0' if you want whole packet to be copied to userspace, which is mostly what you need.
Queue threshold. If a packet is matched by the iptables rule, and already N packets are in the queue, the queue is flushed to userspace. You can use this to implement a policy like: Use a big queue in order to gain high performance, but still have certain packets logged immediately to userspace.
A string that is associated with every packet logged by this rule. You can use this option to later tell from which rule the packet was logged.
The ipt_ULOG kernel module has a couple of module loadtime parameters which can (and should) be tuned to accomodate the needs of the application:
Netlink buffer size. A buffer of the specified size N is allocated for every netlink group that is used. Please note that due to restrictions of the kernel memory allocator, we cannot have a buffer size > 128kBytes. Larger buffer sizes increase the performance, since less kernel/userspace context switches are needed for the same amount of packets. The backside of this performance gain is a potentially larger delay. The default value is 4096 bytes, which is quite small.
The flushtimeout determines, after how many clock ticks (on alpha: 1ms, on x86 and most other platforms: 10ms time units) the buffer/queue is to be flushed, even if it is not full. This can be used to have the advantage of a large buffer, but still a finite maximum delay introduced. The default value is set to 10 seconds.
modprobe ipt_ULOG nlbufsiz=65535 flushtimeout=100
specter is what this is all about, so let's describe it's configuration...
specter reads its configuration parameters from file, which is mostly
`/etc/specter.conf'. It is divided into blocks. Each block start with
a opening curly bracket { and end with closing curly bracket
}. Nesting of blocks (opening new block inside another) is
forbidden, and there's no need for that in specter configuration. In
order to distinguish between blocks, each has a name.
Allowed names for blocks are numbers within range 1-32,
which describe different options for given netlink group, and a special
name global, which is used to specify general daemon parameters.
You cannot define the same block twice, but don't have to define all of
them. In most configurations you'll be fine with two or three blocks.
Each block have to start in a new line, then goes its name and opening bracket. Inside block you specify options, each in a separate line, and end a block with a closing bracket. So, in general, block definition looks like this:
name {
option value
# comment
option "long value that needs spaces"
option
option value # another comment
...
}
As you can see, not every option needs a value, in that case its
presence will override a default (see below for specific options
description). A hash #
is used as a comment, as it will cause a rest of line to be ignored. Of
course you can use comments everywhere, not only inside blocks. If you
need to set an option to a string containing spaces or tabs, you can
enclose it inside double quotation marks, as shown above. And if you
ever manage to write a very long config line, you can cut it by
\ and continue your statement in the line below.
Available global options are:
This options causes specter to continue running despite of errors generated by plugins. That doesn't affect initialization process, when all errors cause an exit. This option can be useful on heavy-load systems, when you expect some malloc() to fail. It doesn't take any arguments.
Path to a file you want specter messages to get logged to. Can be set to
stdout or stderr
The lower the value, the more information is logged. If you experience any problems, check lowest, debug loglevel=1, so that you can see all messages. The highest loglevel is 8, which cause only fatal errors to be shown. The default is 3.
Size of the netlink socket receive memory. You should set this to at least the size of the kernel buffer (nlbufsiz parameter of the ipt_ULOG module). Please note that there is a maximum limit in /proc/sys/net/core/rmem_max which you cannot exceed by increasing the ``rmem'' parameter. You may need to raise the system-wide maximum limit before. You can define this variable in kilobytes (suffix it by 'K') or in megabytes (use 'M' suffix).
Size of the receive buffer. You should set this to at least the size rmem option has. Like rmem can be suffixed by 'M' or 'K'.
Apart from the configfile, there are a couple of commandline options to specter:
Print a help message about the commandline options.
Print version information about specter.
For off into daemon mode. Unless you are debugging, you will want to use this most of the time.
Using this commandline option, an alternate config file can be used. This is important if multiple instances of specter are to be run on a single machine.
This option tells specter to drops its privileges and run as given user.
specter does nearly nothing on its own, it uses plugins for all the dirty work. They are divided into two groups. Input plugins analyze a packet and create hash table concerning received data, in the form like key=value. They don't open files nor they take any input from user. Only output plugins take options. They actually use data from input plugins - save it into logs/databases or execute appropriate commands. So it's vital for you to learn about their configuration, because it's the essence of using specter.
Any option can appear only once in a single block, but there's obviously
an exception. There's one option that can appear in a different blocks
many times. That's of course plugin statement. As an argument you
have to specify full path to the plugin binary, which will mostly be
`/lib/specter/specter_PLUGINNAME.so'.
specter comes with the following input plugins:
Basic input plugin for nfmark, timestamp, mac address, ip header, tcp header, udp header, icmp header, ah/esp header... Most output plugins need this very important plugin.
Example input plugin to log plaintext passwords as used with FTP and POP3. Don't blame me for writing this plugin! The protocols are inherently insecure, and there are a lot of other tools for sniffing passwords... it's just an example.
This is a 'virtual interpreter'. It doesn't really return any information on the packet itself, rather the local system time and hostname. Please note that the time is the time at the time of logging, not the packets receive time.
specter comes with the following output plugins:
This plugin executes specified command when packet is received. By proper use of its functions you can dynamically change your firewall configuration, or even set up simple port-knocking utility.
That option defines a command that should be executed. Don't rely on your $PATH environment variable, and provide full path to an executable. Few printf-like macros can be used, which are expanded during parsing of every packet:
interface packet got received from
interface packet is going to be sent to
IP address of source host
IP address of destination host
IP protocol number (see /etc/protocols)
TCP/UDP source port
TCP/UDP destination port
ICMP type value
When a macros expansion is being done, and any field is empty, executing of a given command is aborted. For example, if you have %i in your execstr and specter gets a tcp packet, command won't be executed, 'cos given macro cannot be expanded (there's no ICMP type field in a TCP packet). You can override this behavior by setting execforce option. Instead of bogus data, string "invalid" will be placed. It's up to executed application to work with that.
If this options is set, daemon will wait until application terminates. It's probably not a good idea to actually use it. If you definitely need it, do it with caution, because it can freeze the whole daemon. Enforcing execution limits should be set in iptables rules by use of limit module, for example.
A very simple output module, dumping all packets in the format
===>PACKET BOUNDARY
key=value
key=value
...
===>PACKET BOUNDARY
...
The module defines the following configuration directives:
The filename where it should log to. The default is
/var/log/specter.oprint
An output module which tries to emulate the old syslog-based LOG targed as far as possible. Logging is done to a seperate textfile instead of syslog, though.
The module defines the following configuration directives:
The filename where it should log to. The default is
/var/log/specter.logemu
Define this option if you want to have your logfile written synchronously. This may reduce performance, but makes your log-lines appear immediately.
An output plugin for logging into a mysql database. This is only compiled if you have the mysql libraries installed, and the configure script was able to detect them. (that is: --with-mysql was specified for ./configure)
The plugin automagically inserts the data into the configured table; It connects to mysql during the startup phase of specter and obtains a list of the columns in the table. Then it tries to resolve the column names against keys of input plugins. This way you can easly select which information you want to log - just by the layout of the table.
If, for example, your table contains a field called 'ip_saddr', specter will resolve this against the key 'ip.saddr' and put the ip address as 32bit unsigned integer into the table.
You may want to have a look at the file 'doc/mysql.table' as an
example table including fields to log all keys from specter_BASE.so. Just
delete the fields you are not interested in, and create the table.
The module defines the following configuration directives:
Name of the mysql database
Name of the table to which specter should log
Name of the mysql database host
Name of the mysql user
Password for mysql
An output plugin for logging into a postgresql database. This is only compiled if you have the postresql libraries installed, and the configure script was able to detect them. (that is: --with-pgsql was specified for ./configure)
The plugin automagically inserts the data into the configured table; It connects to postgresql during the startup phase of specter and obtains a list of the columns in the table. Then it tries to resolve the column names against keys of input plugins. This way you can easly select which information you want to log - just by the layout of the table.
If, for example, your table contains a field called 'ip_saddr', specter will resolve this against the key 'ip.saddr' and put the ip address as 32bit unsigned integer into the table.
You may want to have a look at the file 'doc/pgsql.table' as an
example table including fields to log all keys from specter_BASE.so. Just delete
the fields you are not interested in, and create the table.
The module defines the following configuration directives:
Name of the postgresql database
Name of the table to which specter should log
Name of the postgresql database host
Name of the postgresql user
Password for postgresql
An output plugin that can be used to generate libpcap-style packet logfiles. This can be useful for later analysing the packet log with tools like tcpdump or ethereal.
The module defines the following configuration directives:
The filename where it should log to. The default is:
/var/log/specter.pcap
Set this option if you want to have your pcap logfile written synchronously. This may reduce performance, but makes your packets appear immediately in the file on disk.
This plugin behaves much like LOGEMU, but logs its input into syslog.
Two options are allowed:
Facility a message should be logged with. See syslog(3) manual page. specter accepts following facilities: deamon kernel, user, localx (where x is from 0 to 7).
Importance of a message. All standard syslog levels are allowed: emerg, alert, crit, err, warning, notice, info, debug.
Comments / questions / ... are all welcomed.
Just drop me a note to ruby@joker.linuxstuff.pl.
If an error doesn't happen during compilation time, you are encoured to
get from specter as many information as you can. To do that configure it
with --enable-debug option enabled, and set loglevel (in global
options) to 1. Include information about your system (architecture,
libraries) and a description to help me in reproducing this bug.