STORAGE.CONF(5)STORAGE.CONF(5)NAMEstorage.conf - configuration file for storage manager
DESCRIPTION
The storage manager is a unified interface between INN and
a variety of different storage method, allowing the news
administrator to choose between different storage method
with different tradeoffs (or even use several at the same
time for different newsgroups or articles of different
sizes). The rest of INN need not care what type of stor
age method was used for a given article; the storage man
ager will figure this out automatically when that article
is retrieved via the storage API.
<pathetc in inn.conf>/storage.conf file contains the rules
to be used in assigning articles to different storage
methods.
The file consists of a series of storage method entries.
Blank lines and lines beginning with a number sign (``#'')
are ignored. The maximum number of character in each line
is 255. The order of entries in this file is important.
Each entry specifies a storage method and a set of rules.
Articles that match all of the rules of a storage method
entry will be stored using that storage method. If an
article matches multiple storage method entries, the first
one will be used. Each entry is formatted as follows:
method <methodname> {
class: <storage_class>
newsgroups: <wildmat>
size: <minsize>[,<maxsize>]
expires: <mintime>[,<maxtime>]
options: <options>
}
If spaces or tabs are included in a value, that value must
be quoted with ``"''. If either ``#'' or ``"'' are meant
to be included verbatim in a value, they should be escaped
with ``\''.
<methodname> is the name of a storage method to use for
articles that match the rules of this entry. The cur
rently available storage methods are ``timecaf'', ``time
hash'', ``cnfs'', ``tradspool'' and ``trash''. See the
STORAGE METHODS section below for more details.
The meanings of the keys in each entry are as follows:
class An identifier for this storage method entry.
<storage_class> should be a number and should be
unique across all of the entries in this file.
It's used mainly by expire.ctl(5) for specifying
expiration times by storage class.
newsgroups
What newsgroups are stored using this storage
method. <wildmat> is a wildmat(3) pattern that is
matched against the newsgroups an article is posted
to. If ``storeonxref'' in inn.conf is ``true'',
this pattern will be matched against the newsgroup
names in the ``Xref'' header; otherwise, it will be
matched against newsgroup names in the ``News
groups'' header. Poison wildmat expressions
(expressions starting with ``@'') are allowed and
can be used to exclude certain group patterns.
``!'' cannot be used, however. The <wildmat> pat
tern is matched in order. There is no default
newsgroups pattern; if an entry should match all
newsgroups, use an explicit ``newsgroups: *''.
size A range of article sizes (in bytes) that should be
stored using this storage method. If <maxsize> is
``0'' or not given, the upper size of articles is
limited only by ``maxartsize'' in inn.conf(5). The
``size'' field is optional and may be omitted
entirely if you want articles of any size (that
otherwise fulfill the requirements of this storage
method entry) to be stored in this storage method.
expires
A range of article expiration times that should be
stored using this storage method. Be careful; this
is less useful than it may appear at first. This
is based only on the ``Expires'' header of the
article, not on any local expiration policies or
anything in expire.ctl(5)! If <mintime> is non-
zero, then this entry will not match any article
without an ``Expires'' header. This key is there
fore only really useful for assigning articles with
requested longer expire times to a separate storage
method. <mintime> and <maxtime> are boundaries on
the amount of time into the future the ``Expires''
header of the article requests that it remain
around, and are formatted 0d0h0m0s (days, hours,
minutes, and seconds into the future). If <max
time> is ``0s'' or is not specified, there is no
upper bound on expire times falling into this entry
(note that this key has no effect on when the arti
cle will actually be expired, only on whether or
not the article will be stored using this storage
method). This field is also optional and may be
omitted entirely if all articles with or without an
``Expires'' header (that otherwise fulfill the
requirements of this storage method entry) should
be stored according to it.
options
This key is for passing special options to storage
methods that require them (currently only
``cnfs''). See the STORAGE METHODS section below
for a description of its use.
If an article matches all of the constraints of an entry,
it is stored via that storage method and is associated
with that <storage_class>. This file is scanned in order
and the first matching entry is used to store the article.
If an article doesn't match any entry, either by being
posted to a newsgroup that doesn't match any of the <wild
mat> patterns or by being outside the size and expires
ranges of all entries whose newsgroups pattern it does
match, the article is not stored and is rejected by
innd(8). When this happens, the error message
cant store article: no matching entry in storage.conf
is logged to syslog. If you want to silently drop arti
cles matching certain newsgroup patterns or size or
expires ranges, assign them to the ``trash'' storage
method rather than having them not match any storage
method entry.
STORAGE METHODS
Currently, there are four storage methods available. Each
method has its characteristics. You can choose any of
them to be suitable for your environment. Note that each
method has an attribute ``EXPENSIVESTAT'' which means
whether checking existense of article is expensive or not.
This is used to run expireover(8).
cnfs The ``cnfs'' storage method stores articles in
large cyclic buffers (CNFS stands for Cyclic News
File System). It's by far the fastest of all stor
age methods (except for ``trash''), since it elimi
nates the overhead of dealing with a file system
and creating new files. Articles are stored in
CNFS buffers in arrival order, and when the buffer
fills, it wraps around to the beginning and stores
new articles over top of the oldest articles in the
buffer. The expire time of articles stored in CNFS
buffers is therefore entirely determined by how
long it takes the buffer to wrap around, which
depends on how much data is being stored in it.
(This method is therefore said to have self-expire
functionality.) ``EXPENSIVESTAT'' is ``FALSE'' for
this method. CNFS has its own configuration file,
cycbuff.conf(5). Storage method entries for the
``cnfs'' storage method must have an ``options''
field specifying the metacycbuff into which arti
cles matching that entry should be stored. See
cycbuff.conf(5) for details on metacycbuffs.
timecaf
This method stores multiple articles in one file,
whose name is based on the article's arrival time
and the storage class. The file name will be
<patharticles in inn.conf>/timecaf-nn/bb/aacc.CF,
where ``nn'' is the hexadecimal value of <stor
age_class> and ``bb'' and ``aacc'' are hexadecimal
components of the arrival time. (The arrival time
in seconds since epoch is converted to hex and
interpreted as 0xaabbccFF, with ``aa'', ``bb'', and
``cc'' used to build the path.) This method does
not have self-expire functionality (meaning
expire(8) has to run periodically to delete old
articles). ``EXPENSIVESTAT'' is ``FALSE'' for this
method.
timehash
This method is very similar to ``timecaf'' except
that each article is stored in a separate file.
The name of the file for a given article will be
<patharticles in inn.conf>/time-nn/bb/cc/yyyy-aadd,
where ``nn'' is the hexadecimal value of <stor
age_class>, ``yyyy'' is a hexadecimal sequence num
ber, and ``bb'', ``cc'', and ``aadd'' are compo
nents of the arrival time in hexadecimal (the
arrival time is converted to hex and interpreted as
0xaabbccdd). This method does not have self-expire
functionality. ``EXPENSIVESTAT'' is ``TRUE'' for
this method.
tradspool
Traditional spool, or ``tradspool'', is the tradi
tional news article storage format. Each article
is stored in a file named: <patharti_
cles in inn.conf>/news/group/name/nnnnn, where
``news/group/name'' is the name of the newsgroup to
which the article was posted with each period
changed to a slash, and ``nnnnn'' is the sequence
number of the article in that newsgroup. For
crossposted articles, the article is linked into
each newsgroup to which it is crossposted (using
either hard or symbolic links). This is the way
all versions of INN prior to 2.0 stored all arti
cles, as well as being the article storage format
used by C News and earlier news systems. This
method does not have self-expire functionality.
``EXPENSIVESTAT'' is ``TRUE'' for this method.
trash This method silently discards all articles stored
in it. Its only real uses are for testing and for
silently discarding articles matching a particular
storage method entry (for whatever reason). Arti
cles stored in this method take up no disk space
and can never be retrieved, so this method has
self-expire functionality of a sort. ``EXPENSIVES
TAT'' is ``FALSE'' for this method.
EXAMPLE
The following sample storage.conf file would store all
articles posted to alt.binaries.* in the ``BINARIES'' CNFS
metacycbuff, all articles over roughly 50KB in any other
hierarchy in the ``LARGE'' CNFS metacycbuff, all other
articles in alt.* in one timehash class, and all other
articles in any newsgroups in a second timehash class,
except for the internal.* hierarchy which is stored in
traditional spool format.
method tradspool {
class: 1
newsgroups: internal.*
}
method cnfs {
class: 2
newsgroups: alt.binaries.*
options: BINARIES
}
method cnfs {
class: 3
newsgroups: *
size: 50000
options: LARGE
}
method timehash {
class: 4
newsgroups: alt.*
}
method timehash {
class: 5
newsgroups: *
}
Notice that the last storage method entry will catch
everything. This is a good habit to get into; make sure
that you have at least one catch-all entry just in case
something you didn't expect falls through the cracks.
Notice also that the special rule for the internal.* hier
archy is first, so it will catch even articles crossposted
to alt.binaries.* or over 50KB in size.
HISTORY
Written by Katsuhiro Kondou <kondou@nec.co.jp> for Inter
NetNews. This is revision 1.7.2.2, dated 2001/04/17.
SEE ALSOcycbuff.conf(5), expire.ctl(5), inn.conf(5), innd(8),
newsfeeds(5), wildmat(3).
STORAGE.CONF(5)
