STORAGE.CONF(5)STORAGE.CONF(5)NAMEstorage.conf - configuration file for storage manager
DESCRIPTION
The storage manager is a unified interface between INN and a variety of
different storage method, allowing the news administrator to choose
between different storage methods with different tradeoffs (or even use
several at the same time for different newsgroups, or articles of dif‐
ferent sizes). The rest of INN need not care what type of storage
method was used for a given article; the storage manager will figure
this out automatically when that article is retrieved via the storage
API.
The <pathetc in inn.conf>/storage.conf file contains the rules to be
used in assigning articles to different storage methods.
The file consists of a series of storage method entries. Blank lines
and lines beginning with a number sign (``#'') are ignored. The maxi‐
mum number of characters in each line is 255. The order of entries in
this file is important, see below.
Each entry specifies a storage method and a set of rules. Articles
that match all of the rules of a storage method entry will be stored
using that storage method; if an article matches multiple storage
method entries, the first will be used. Each entry is formatted as
follows:
method <methodname> {
class: <storage_class>
newsgroups: <wildmat>
size: <minsize>[,<maxsize>]
expires: <mintime>[,<maxtime>]
options: <options>
exactmatch: <bool>
}
If spaces or tabs are included in a value, that value must be quoted
with ``"''. If either ``#'' or ``"'' are meant to be included verbatim
in a value, they should be escaped with ``\''.
<methodname> is the name of a storage method to use for articles that
match the rules of this entry. The currently available storage methods
are ``timecaf'', ``timehash'', ``cnfs'', ``tradspool'' and ``trash''.
See the STORAGE METHODS section below for more details.
The meanings of the keys in each entry are as follows:
class An identifier for this storage method entry. <storage_class>
should be a number and should be unique across all of the
entries in this file. It's used mainly for specifying expira‐
tion times by storage class as described in expire.ctl(5).
newsgroups
What newsgroups are stored using this storage method. <wildmat>
is a uwildmat(3) pattern that is matched against the newsgroups
an article is posted to. If ``storeonxref'' in inn.conf is
``true'', this pattern will be matched against the newsgroup
names in the ``Xref'' header; otherwise, it will be matched
against newsgroup names in the ``Newsgroups'' header (see
inn.conf(5) for discussion of the differences between these pos‐
sibilities). Poison wildmat expressions (expressions starting
with ``@'') are allowed and can be used to exclude certain group
patterns. ``!'' cannot be used, however. The <wildmat> pattern
is matched in order. There is no default newsgroups pattern; if
an entry should match all newsgroups, use an explicit ``news‐
groups: *''.
size A range of article sizes (in bytes) that should be stored using
this storage method. If <maxsize> is ``0'' or not given, the
upper size of articles is limited only by ``maxartsize'' in
inn.conf. The ``size'' field is optional and may be omitted
entirely if you want articles of any size (that otherwise ful‐
fill the requirements of this storage method entry) to be stored
in this storage method.
expires
A range of article expiration times that should be stored using
this storage method. Be careful; this is less useful than it
may appear at first. This is based only on the ``Expires''
header of the article, not on any local expiration policies or
anything in expire.ctl! If <mintime> is non-zero, then this
entry will not match any article without an ``Expires'' header.
This key is therefore only really useful for assigning articles
with requested longer expire times to a separate storage method.
Articles only match if the time until expiration (that is, the
amount of time into the future that the ``Expires'' header of
the article requests that it remain around) falls in the inter‐
val specified by <mintime> and <maxtime>. The format of these
parameters is 0d0h0m0s (days, hours, minutes, and seconds into
the future). If <maxtime> is ``0s'' or is not specified, there
is no upper bound on expire times falling into this entry (note
that this key has no effect on when the article will actually be
expired, only on whether or not the article will be stored using
this storage method). This field is also optional and may be
omitted entirely if all articles with or without an ``Expires''
header (that otherwise fulfill the requirements of this storage
method entry) should be stored according to it.
options
This key is for passing special options to storage methods that
require them (currently only ``cnfs''). See the STORAGE METHODS
section below for a description of its use.
exactmatch
If this key is set to ``true'', all newsgroups will be examined
to see if they match newsgroups patterns. (Normally, any
nonzero number of matching newsgroups is sufficient, provided no
newsgroup matches a poison wildmat as described above.) This is
a boolan value and ``true'', ``yes'' and ``on'' are usable to
enable this key. The case of these values is not significant.
The default is false.
If an article matches all of the constraints of an entry, it is stored
via that storage method and is associated with that <storage_class>.
This file is scanned in order and the first matching entry is used to
store the article.
If an article doesn't match any entry, either by being posted to a
newsgroup that doesn't match any of the <wildmat> patterns or by being
outside the size and expires ranges of all entries whose newsgroups
pattern it does match, the article is not stored and is rejected by
innd(8). When this happens, the error message
cant store article: no matching entry in storage.conf
is logged to syslog. If you want to silently drop articles matching
certain newsgroup patterns or size or expires ranges, assign them to
the ``trash'' storage method rather than having them not match any
storage method entry.
STORAGE METHODS
Currently, there are four storage methods available. Each method has
its pros and cons; you can choose any mixture of them as is suitable
for your environment. Note that each method has an attribute ``EXPEN‐
SIVESTAT'' which indicates whether checking the existence of an arti‐
cle is expensive or not. This is used to run expireover(8).
cnfs The ``cnfs'' storage method stores articles in large cyclic buf‐
fers (CNFS stands for Cyclic News File System). It's by far the
fastest of all storage methods (except for ``trash''), since it
eliminates the overhead of dealing with a file system and creat‐
ing new files. Articles are stored in CNFS buffers in arrival
order, and when the buffer fills, it wraps around to the begin‐
ning and stores new articles over top of the oldest articles in
the buffer. The expire time of articles stored in CNFS buffers
is therefore entirely determined by how long it takes the buffer
to wrap around, which depends on how quickly data is being
stored in it. (This method is therefore said to have self-
expire functionality.) ``EXPENSIVESTAT'' is ``false'' for this
method. CNFS has its own configuration file, cycbuff.conf,
which describes some subtlties to the basic description given
above. Storage method entries for the ``cnfs'' storage method
must have an ``options'' field specifying the metacycbuff into
which articles matching that entry should be stored; see
cycbuff.conf(5) for details on metacycbuffs.
timecaf
This method stores multiple articles in one file, whose name is
based on the article's arrival time and the storage class. The
file name will be <patharticles in inn.conf>/timecaf-
nn/bb/aacc.CF, where ``nn'' is the hexadecimal value of <stor‐
age_class>, ``bb'' and ``aacc'' are hexadecimal components of
the arrival time, and ``CF'' is a hardcoded extension. (The
arrival time, in seconds since the epoch, is converted to hexa‐
decimal and interpreted as 0xaabbccdd, with ``aa'', ``bb'', and
``cc'' used to build the path.) This method does not have self-
expire functionality (meaning expire(8) has to run periodically
to delete old articles). ``EXPENSIVESTAT'' is ``false'' for
this method.
timehash
This method is very similar to ``timecaf'' except that each
article is stored in a separate file. The name of the file for
a given article will be <patharticles in inn.conf>/time-
nn/bb/cc/yyyy-aadd, where ``nn'' is the hexadecimal value of
<storage_class>, ``yyyy'' is a hexadecimal sequence number, and
``bb'', ``cc'', and ``aadd'' are components of the arrival time
in hexadecimal (the arrival time is interpreted as documented
above under ``timecaf''). This method does not have self-expire
functionality. ``EXPENSIVESTAT'' is ``true'' for this method.
tradspool
Traditional spool, or ``tradspool'', is the traditional news
article storage format. Each article is stored in a file named:
<patharticles in inn.conf>/news/group/name/nnnnn, where
``news/group/name'' is the name of the newsgroup to which the
article was posted with each period changed to a slash, and
``nnnnn'' is the sequence number of the article in that news‐
group. For crossposted articles, the article is linked into
each newsgroup to which it is crossposted (using either hard or
symbolic links). This is the way versions of INN prior to 2.0
stored all articles, as well as being the article storage format
used by C News and earlier news systems. This method does not
have self-expire functionality. ``EXPENSIVESTAT'' is ``true''
for this method.
trash This method silently discards all articles stored in it. Its
only real uses are for testing and for silently discarding arti‐
cles matching a particular storage method entry (for whatever
reason). Articles stored in this method take up no disk space
and can never be retrieved, so this method has self-expire func‐
tionality of a sort. ``EXPENSIVESTAT'' is ``false'' for this
method.
EXAMPLE
The following sample storage.conf file would store all articles posted
to alt.binaries.* in the ``BINARIES'' CNFS metacycbuff, all articles
over roughly 50 KB in any other hierarchy in the ``LARGE'' CNFS metacy‐
cbuff, all other articles in alt.* in one timehash class, and all other
articles in any newsgroups in a second timehash class, except for the
internal.* hierarchy which is stored in traditional spool format.
method tradspool {
class: 1
newsgroups: internal.*
}
method cnfs {
class: 2
newsgroups: alt.binaries.*
options: BINARIES
}
method cnfs {
class: 3
newsgroups: *
size: 50000
options: LARGE
}
method timehash {
class: 4
newsgroups: alt.*
}
method timehash {
class: 5
newsgroups: *
}
Notice that the last storage method entry will catch everything. This
is a good habit to get into; make sure that you have at least one
catch-all entry just in case something you didn't expect falls through
the cracks. Notice also that the special rule for the internal.* hier‐
archy is first, so it will catch even articles crossposted to alt.bina‐
ries.* or over 50 KB in size.
HISTORY
Written by Katsuhiro Kondou <kondou@nec.co.jp> for InterNetNews. This
is revision 6124, dated 2003-01-14.
SEE ALSOcycbuff.conf(5), expire.ctl(5), inn.conf(5), innd(8), newsfeeds(5),
uwildmat(3).
STORAGE.CONF(5)
