Upgrading third-party applications to work with ClamAV 0.95

Upgrading to the new libclamav API

The API of libclamav has changed in 0.95, the new API should allow future extensions to be made to the API without breaking clients that use it. However clients (third-party applications) that use the 0.94 API should be upgraded to the 0.95 API, otherwise they won't work.

For more details see:

• Section 6 of clamdoc.pdf
• example on how to use libclamav API
• the bugzilla entry for the API change

Here is a table summarizing the changes between libclamav API 0.94 and 0.95:

Other changes:

• struct cl_engine is no longer defined in clamav.h, allowing us to change its internal structure without breaking the API/ABI in the future
• error codes are now part of an enum, thus error codes have a positive value now. You should treat anything different than CL_SUCCESS/CL_CLEAN as an error, and call cl_strerror() to find out what the failure was.

Upgrading to the new clamd protocol extensions

Clamd 0.95 introduces new protocol commands, but is backwards compatible with old clients, with the exception of the SESSION command, which got removed in favour of the IDSESSION command.

Upgrading clamd clients that don't use SESSION is not required, they should continue to work with the old commands. If they don't please open a bugreport.

For more details see:

• Section 5.1 of clamdoc.pdf

In clamd 0.95 it is recommended to prefix all commands with the z character, to indicate that the command will be deliminated by a '\0' character; clamd will then continue reading until a '\0' character is read. This is similar to the n prefix that was supported in 0.94 too, and which is still accepted (indicating a \n terminator). Clamd honours the terminator in its replies.

If a command is not recognized, or doesn't follow the protocol, clamd will reply with an error message, and close the connection: clamd may now give an error message and close the connection, instead of just closing the connection.

Changes:

• RELOAD - will reload the database immediately (and accept new connections while doing that), instead of reloading when next command is received, and blocking new connections
• STREAM - deprecated (but still accepted). See the new INSTREAM command.
• SESSION - removed, you should switch to using IDSESSION
• FILDES - it existed in 0.94, but wasn't documented in clamdoc.pdf.

New commands:

• INSTREAM - sends data to be scanned on the same connection, avoiding the overhead of a new TCP connection and problems with NAT.
  • It is mandatory to prefix this command with n or z. The stream is sent to clamd in chunks, after INSTREAM, on the same socket on which the command was sent. This avoids the overhead of establishing new TCP connections and problems with NAT.
  • The format of the chunk is: <length><data> where <length> is the size of the following data in bytes expressed as a 4 byte unsigned integer in network byte order and <data> is the actual chunk.
  • Streaming is terminated by sending a chunk, where length is zero.
  • Note: do not exceed StreamMaxLength as defined in clamd.conf, otherwise clamd will reply with INSTREAM size limit exceeded and close the connection.

• STATS - query clamd about statistics on queue and memory usage. The exact format of the reply is subject to change in the future.
• IDSESSION, END - issue multiple commands on the same connection, replies are asynchronous (as soon as a scan has finished), unlike with the old SESSION command, where commands were processed sequentially. Replies now have an <id>:   prefix to differentiate between replies for each command, for example: "1: OK".
  • It is mandatory to prefix IDSESSION, and all commands inside it with either an n or a z, otherwise clamd won't recognize the command.
  • Within a session, multiple SCAN, INSTREAM, FILDES, VERSION, STATS commands can be sent on the same socket without opening a new connection. Replies from clamd will be in the form <id>: <response> where <id> is the request number (in ASCII, starting from 1) and <response> is the usual clamd reply. The reply lines have the same \n or \0 delimiter as the corresponding command. Clamd will process the commands asynchronously, and reply as soon as it has finished processing.
  • Clamd requires clients to read available replies, before sending more commands to prevent send() deadlocks. The recommended way to implement a client that uses IDSESSION is with non-blocking sockets, and a select()/poll() loop: whenever send() would block, sleep in select()/poll() until either you can write more data, or read more replies.
  • Note that using non-blocking sockets without the select/poll loop and alternating recv()/send() doesn’t comply with clamd’s requirements. If clamd detects that a client has dead-locked, it will close the connection. Note that clamd may close an IDSESSION connection too if the client doesn’t follow the protocol’s requirements.

Using the new clamav-milter

The clamav-milter in 0.95 has been completely rewritten with a minimalistic approach. Clamav-milter now cannot stand on its own (formerly known as internal mode) but requires at least one clamd to operate (old external mode). Clamav-milter has now got its own configuration file. Let's take a look at its options:

The first thing we need is an interface socket to Sendmail/Postfix:

# Define the interface through which we communicate with sendmail
# This option is mandatory! Possible formats are:
# [[unix|local]:]/path/to/file - to specify a unix domain socket
# inet:port@[hostname|ip-address] - to specify an ipv4 socket
# inet6:port@[hostname|ip-address] - to specify an ipv6 socket
#
# Default: no default
#MilterSocket /tmp/clamav-milter.socket
#MilterSocket inet:7357

Then we need a second socket to interface to clamd:

# Define the clamd socket to connect to for scanning.
# This option is mandatory! Syntax:
# ClamdSocket unix:path
# ClamdSocket tcp:host:port
# The first syntax specifies a local unix socket (needs an bsolute path) e.g.:
#     ClamdSocket unix:/var/run/clamd/clamd.socket
# The second syntax specifies a tcp local or remote tcp socket: the
# host can be a hostname or an ip address; the ":port" field is only required
# for IPv6 addresses, otherwise it defaults to 3310
#     ClamdSocket tcp:192.168.0.1
#
# This option can be repeated several times with different sockets or even
# with the same socket: clamd servers will be selected in a round-robin fashion.
#
# Default: no default
#ClamdSocket tcp:scanner.mydomain:7357

Clamav-milter does not need to be run as a privileged user or group. You can fiddle with the following options or leave them unset to let it run as the current user:

# Run as another user (clamav-milter must be started by root for this option to work)
#
# Default: unset (don't drop privileges)
#User clamav

# Initialize supplementary group access (clamav-milter must be started by root).
#
# Default: no
#AllowSupplementaryGroups no

Optionally, clamav-milter can be run inside a chroot. If you do set this option, please ensure you understand implication for paths and privileges.

# Chroot to the specified directory.
# Chrooting is performed just after reading the config file and before dropping privileges.
#
# Default: unset (don't chroot)
#Chroot /newroot

Then we have the actions configuration. Here's how to tell the milter what to do with the emails:

# The following group of options controls the delievery process under
# different circumstances.
# The following actions are available:
# - Accept
#   The message is accepted for delievery
# - Reject
#   Immediately refuse delievery (a 5xx error is returned to the peer)
# - Defer
#   Return a temporary failure message (4xx) to the peer
# - Blackhole (not available for OnFail)
#   Like accept but the message is sent to oblivion
# - Quarantine (not available for OnFail)
#   Like accept but message is quarantined instead of being delivered
#   In sendmail the quarantine queue can be examined via mailq -qQ
#   For Postfix this causes the message to be accepted but placed on hold
# 
# Action to be performed on clean messages (mostly useful for testing)
# Default Accept
#OnClean Accept

# Action to be performed on infected messages
# Default: Quarantine
#OnInfected Quarantine

# Action to be performed on error conditions (this includes failure to
# allocate data structures, no scanners available, network timeouts,
# unknown scanner replies and the like)
# Default Defer
#OnFail Defer

Clamav-milter knows two type of exclusions to the above rules.
Firstly, exclusions based on the originating host/network:

# Messages originating from these hosts/networks will not be scanned
# This option takes a host(name)/mask pair in CIRD notation and can be
# repeated several times. If "/mask" is omitted, a host is assumed.
# To specify a locally orignated, non-smtp, email use the keyword "local"
#
# Default: unset (scan everything regardless of the origin)
#LocalNet local
#LocalNet 192.168.0.0/24
#LocalNet 1111:2222:3333::/48

Secondly, exclusions based on the sender's or recipient's email address:

# This option specifies a file which contains a list of POSIX regular
# expressions. Addresses (sent to or from - see below) matching these regexes
# will not be scanned.  Optionally each line can start with the string "From:"
# or "To:" (note: no whitespace after the colon) indicating if it is, 
# respectively, the sender or recipient that is to be whitelisted.
# If the field is missing, "To:" is assumed.
# Lines starting with #, : or ! are ignored.
#
# Default unset (no exclusion applied)
#Whitelist /etc/whitelisted_addresses

Several options are provided to tune the logging (note that clamav-milter now has its own log files), the most important are:

# Uncomment this option to enable logging.
# LogFile must be writable for the user running daemon.
# A full path is required.
#
# Default: disabled
#LogFile /tmp/clamav-milter.log

# Use system logger (can work together with LogFile).
#
# Default: no
#LogSyslog yes

The remaining options are extensively documented inside the example config file.

Additionally a quick and dirty Perl script is available to automatically generate a clamav-milter.conf file. Just invoke the script with the same parameters you were passing to the old milter and then review all the preset options to make sure everything is sane.

For more information, see this post on the mailing list and the ClamAV blog.

Packaging and Dependencies

• The dependency on LibGMP has been dropped, libclamav now bundles LibTomMath
  • this means you no longer have to install libgmp-dev to compile ClamAV
  • package maintainers: if you want to use the system's libtommath, instead of the bundled one use ./configure --with-system-tommath, in that case configure will search for libtommath in the usual places for system libraries (/usr/lib etc.). If it can't find it, it will fallback to the bundled version. However, look out for broken versions of compilers/runtimes if you use the system's libtommath, they have been worked around in the bundled version.
• ClamAV now bundles a version of libltdl (from libtool-2.2.6a), configure will automatically pick the system's ltdl if the version is new enough
  • this means that if you build ClamAV without UNRAR support, and you later install libclamunrar_iface, then the library will automatically be picked up, and unrar support will be available
  • for example, you can provide a DFSG-free packaged version of ClamAV without UNRAR, which will allow your users to later install a nonfree libclamunrar and have everything working
    • The license of libclamunrar_iface is LGPL, see COPYING.unrar in the source distribution for the license of libclamunrar
• be aware that the new milter doesn't require to be run as a privileged user, startup scripts should be modified accordingly
  • when used with Postfix, make sure that the socket that the milter creates is accessible by Postfix (correct user:group and permissions), previously this was not an issue, since root ignores permissions
• be aware that the new milter doesn't accept the old command line options anymore, see "Using the new clamav-milter" above on how to convert the command line options to a configuration file
• it is recommended that you integrate 'make check' into your build process to ensure that the binaries work on all the architectures for which you build the package
• if ncurses (or pdcurses) is available, then clamdtop will be built too

Caveats

• If you see this error message: error while loading shared libraries: libclamav.so.6: cannot open shared object file: No such file or directory then you should run ldconfig
• You may see this error: LibClamAV Warning: Cannot dlopen: file not found - unrar support unavailable
  • if you run make check without uninstalling 0.94.2 first, uninstalling 0.94.2 should solve the problem
  • Unter RedHat 5.3, 64Bit with RPMs from http://packages.sw.be/clamav/ : "cd /usr/lib64; ln -s libclamunrar_iface.so.6 libclamunrar_iface.so; ldconfig" will resolve this issue
  • on Solaris if you install ClamAV into a custom prefix that is not on the runtime linker's path: use crle to add the path where you installed libclamav to the linker's path, or set LD_LIBRARY_PATH

-- TorokEdwin - 27 Feb 2009