The main.cvd updates performed in 2005 and 2006 caused an enormous traffic on many database mirrors (some of them hit 100 mbit/s) and showed that the previous incremental update scheme based on two database files (main.cvd and daily.cvd) is no longer sufficient.

Previous situation

Sigmakers update the daily.cvd database usually a few times a day. When the database becomes too big (>­1MB) most of the signatures get moved to the main database. With the current number of ClamAV users and the size of main.cvd almost 4MB (as of May, 8th 2006) every main update generates a huge load on the mirrors and makes the admins unhappy.

Current situation

The current update model was a big success and what's most important it was an efficient and very stable solution. To preserve backward compatibility and make the new procedure safe (i.e. allow fallback to the old method in case of failure) it will remain untouched and the new model will use CVD files as a starting point.

The basic method is to unpack CVD files into a temporary subdirectory of the standard database directory, apply the incremental updates on these, create a CVD style database file from that directory and remove the temporary directory.

The database file will be compressed if the freshclam option CompressLocalDatabase is set to yes. It will get the suffix .cld to differentiate it from a database file downloaded directly.

Files and data structures

A new file is added to every container file: <dbname>.info (so e.g. in main.cvd there is the file main.info) Its first line is formatted like a CVD header and contains the same information. The other lines enumerate the database files, together with a checksum, separated form the filename by a :

To update a database (main or daily), from version old to version new, the patches named /<dbname>-<n>.cdiff for n from old+1 .. new are fetched from a mirror. The mirrors will keep the last 10 days update scripts to handle clients which call freshclam only once per week or so.

The format of each file is: header : gzip compressed script : base64 encoded signature. The header has the form ClamAV-Diff : version : bytes, where bytes is the length of the uncompressed script. The signature is found by searching for the : in the last 350 bytes of the file.

The script is a line oriented plain text file, each line is of the form command arg 1 .. arg n, the following commands are known:

• OPEN dbase Load a selected database into memory
• ADD sig Add a single signature at the end
• DEL lineno start Remove a single signature from the database
• XCHG lineno start line Replace the content of the specified line
• CLOSE ­Close the current database and save changes to disk
• MOVE src dst startlineno start endlineno start Move signatures from database src to database dst
• UNLINK dbase Delete a database

-- NilsRennebarth - 25 Nov 2008