awffull(1)awffull(1)NAME
AWFFull - A Webalizer Fork, Full o' features
SYNOPSISawffull [...] [log-file]
DESCRIPTION
AWFFull is a web server log analysis program based on The Webalizer.
AWFFull produces usage statistics in HTML format for viewing with a
browser. The results are presented in both columnar and graphical for‐
mat, which facilitates interpretation. Yearly, monthly, daily and
hourly usage statistics are presented, along with the ability to dis‐
play usage by site, URL, referrer, user agent (browser), user name,
search strings, entry/exit pages, and country (some information may not
be available if not present in the log file being processed).
AWFFull supports the following log formats shown in the following vari‐
able list:
CLF (common log format) log files
Combined
log formats as defined by NCSA and others, and variations of
these which it attempts to handle intelligently
xferlog
wu-ftpd formatted log files allowing analysis of ftp servers,
and squid proxy logs.
Note
Logs may also be compressed, via gzip. If a compressed log file
is detected, it will be automatically uncompressed while it is
read. Compressed logs must have the standard gzip extension of
.gz.
This documentation applies to AWFFull Version 3.8.2
CHANGES FROM WEBALIZER
AWFFull is based on The Webalizer code and has a number of large and
small changes. These include:
· Beyond the raw statistics: Making use of published formulae to pro‐
vide additional insights into site usage
· GeoIP IP Address look-ups for more accurate country detection
· Resizable graphs
· Integration with GNU gettext allowing for ease of translations. Cur‐
rently 32 languages are supported.
· Display more than 12 months of the site history on the front page.
· Additional page count tracking and sort by same.
· Some minor visual tweaks, including Geolizer's use of Kb, Mb etc for
Volumes
· Additional Pie Charts for URL counts, Entry and Exit Pages, and Sites
· Horizontal lines on graphs that are more sensible and easier to read
· User Agent and Referral tracking is now calculated via PAGES not HITS
· GNU style long command line options are now supported (eg --help)
· Can choose what is a page by excluding ‘what isn't’ vs the original
‘what is’ method
· Requests to the site being analysed are displayed with the matching
referring URL
· A Table of 404 Errors, and the referring URL can be generated
· An external CSS file can be used with the generated html
· Manual performance optimisation of the config file is now easier with
a post analysis summary output
· Specified IP Addresses can be assigned to a given country
· Additional Dump options for detailed analysis with other tools
· Lotus Domino v6 logs are now detected and processed
Additional changes and improvements are planned and undergoing imple‐
mentation. See the TODO file for details.
NEW REPORT MEASUREMENTS
With version 3.8.1 of AWFFull, several new measured results have been
added to the detailed report monthly page.
Single Access
Single Access Pages - the only page seen within a given visit
Stickiness
How useful a given entry page is to draw Visitors deeper into
your site
Popularity
The Ratio of Page Entries to Page Exits
These metrics can help towards improving insight in the usage of the
processed web site. And hence allow the site owner to make positive
change to make the site more useful to site visitors. All three metrics
appear in the ‘Entry Pages’ Report. ‘Popularity’ is also on the ‘Exit
Pages’ Report.
Single Access
More completely: ‘Single Access Pages’. This is a report on the number
of times that a given page was the only page viewed within a Visit. Or
in English, Someone came to your website. They only viewed one page.
The number is the cumulative count of people who did this for that par‐
ticular page. Why is this useful? Identifying those entry pages that
don't draw visitors deeper into your site. Or seeing entry pages that
shouldn't be entry pages. It's also a reality check against the next
two values which are calculated from this number. The number generated
should be a subset of the ‘Entry Page Views’ and/or ‘Exit Page Views’
metric. If it isn't? Let me know, we have a bug. :-)
Stickiness
Is calculated as 1 - (Single Access / Entry Page Views) expressed as a
percentage. In essence Stickiness describes how useful a given entry
page is to draw Visitors deeper into your site. The stickier the page,
the more folk are caught by it. :-) The closer to 100% the better. Gen‐
erally. Certain pages within YOUR website may not make sense to have a
high stickiness or even > 5%. This measurement is a clue to understand‐
ing how your site is used, it is not a rule. How is this useful? How
and where are people entering your web site. Does that make sense?
Should it be here or there? What can you change to fix this and hence
improve their use of your website.
Popularity
Popularity is the Ratio of Page Entries to Page Exits. o If it equals
1.0? Then the number of visitors to your site who started with that
page, equals the number who left at that page. o If greater then 1.0,
then more people entered here then left. o If less then 0? More people
left from here then entered. I personally find this metric one of the
more useful "At a Glance: How are Pages Performing" metrics. One of the
difficulties with using this particular metric is that certain numbers
will NOT make sense for YOUR site. In that a natural exit page would
expect to have a very low Popularity. It's an exit page, not an entry
page. So if an exit page has a high popularity, then you have a real
problem. Likewise, a low Popularity for an entry page is unlikely to be
a Good Thing(tm).
"Where & Why?" All three of these metrics are covered very nicely in
Hack #58 from "Web Site Measurement Hacks" [1]. Which is where, credit
where credit due, the inspiration to add these metrics came from.
RUNNING AWFFULL
AWFFull is designed to be run from a Unix command line prompt or as a
crond(8) job. There is no need to run with super-user privleges, and
indeed, is preferable NOT to.
Once executed, the general flow of the program is:
A default configuration file is scanned for, /usr/local/etc/awf‐
full.conf and, if found, is used.
Any command line arguments given to the program are parsed. This may
include the specification of one or more configuration files, which are
processed at the time it is encountered.
It can be useful to have multiple config files. A master used for mul‐
tiple sites, and individualised config files. Do be aware that last op‐
tion set wins. So last config file, or if after a config file, command
line options. Useful if you desire to send the output to an alternate
directory.
If a log file was specified, it is opened and made ready for process‐
ing. If no log file was given, STDIN is used for input. If the log
filename '-' is specified, STDIN will be forced.
If an output directory was specified, AWFFull changes to that directory
in preparation for generating output. If no output directory was given,
the current directory is used.
If no hostname was given, the program attempts to get the hostname us‐
ing a uname(2) system call. If that fails, localhost is used.
A history file is searched for in the current directory (output direc‐
tory) and read if found. This file keeps totals for previous months,
which is used in the main index.html HTML document. Note: The file lo‐
cation can now be specified with the HistoryName configuration option.
If incremental processing was specified, a data file is searched for
and loaded if found, containing the 'internal state' data of the pro‐
gram at the end of a previous run. Note: The file location can now be
specified with the IncrementalName configuration option.
Main processing begins on the log file. If the log spans multiple
months, a separate HTML document is created for each month. After main
processing, the main index.html page is created, which has totals by
month and links to each months HTML document.
A new history file is saved to disk, which includes totals generated by
AWFFull during the current run.
If incremental processing was specified, a data file is written that
contains the 'internal state' data at the end of this run.
INCREMENTAL PROCESSING
Version 1.2x of The Webalizer added incremental run capability. Simply
put, this allows processing large log files by breaking them up into
smaller pieces, and processing these pieces instead. What this means in
real terms is that you can now rotate your log files as often as you
want, and still be able to produce monthly usage statistics without the
loss of any detail. Basically, AWFFull saves and restores all internal
data in a file named awffull.current. This allows the program to 'start
where it left off' so to speak, and allows the preservation of detail
from one run to the next. The data file is placed in the current output
directory, and is a plain ASCII text file that can be viewed with any
standard text editor. It's location and name may be changed using the
IncrementalName configuration keyword.
Some special precautions need to be taken when using the incremental
run capability of AWFFull. Configuration options should not be changed
between runs, as that could cause corruption of the internal data
stored. For example, changing the MangleAgents level will cause differ‐
ent representations of user agents to be stored, producing invalid re‐
sults in the user agents section of the report. If you need to change
configuration options, do it at the end of the month after normal pro‐
cessing of the previous month and before processing the current month.
You may also want to delete the awffull.current file as well.
AWFFull also attempts to prevent data duplication by keeping track of
the timestamp of the last record processed. This timestamp is then com‐
pared to current records being processed, and any records that were
logged previous to that timestamp are ignored. This, in theory, should
allow you to re-process logs that have already been processed, or
process logs that contain a mix of processed/not yet processed records,
and not produce duplication of statistics.
The only time this may break is if you have duplicate timestamps in two
separate log files. Any records in the second log file that do have the
same timestamp as the last record in the previous log file processed,
will be discarded as if they had already been processed. There are lots
of ways to prevent this however, for example, stopping the web server
before rotating logs will prevent this situation, or using a tool such
as cronolog (⟨http://cronolog.org/⟩). This setup also necessitates
that you always process logs in chronological order, otherwise data
loss will occur as a result of the timestamp compare.
REVERSE DNS LOOKUPS
AWFFull no longer supports DNS lookups. Please use an external program
such as DNShistory or DNSTran instead.
· ⟨http://www.summary.net/soft/dnstran.html⟩
· ⟨http://www.stedee.id.au/dnshistory⟩
With version 3.7.1 of AWFFull, GeoIP capability can be used for im‐
proved country detection.
COMMAND LINE OPTIONS
AWFFull supports many different configuration options that will alter
the way the program behaves and generates output. Most of these can be
specified on the command line, while some can only be specified in a
configuration file. The command line options are listed below, with
references to the corresponding configuration file keywords. See also
awffull.conf(5).
General Options
-h, --help
Display all available command line options and exit program
-V, --version
Display program version and exit program
-v --verbose
Verbosity Display debugging information for errors and warnings.
Multiple v's will increase the amount of information displayed.
--match_counts
Display optimisation useful information pertaining to the number
of matches against various Group, Hide and Ignore options.
-i --ignore_history
IgnoreHist Ignore history. USE WITH CAUTION. This will cause
AWFFull to ignore any previous monthly history file only. Incre‐
mental data (if present) is still processed.
-p --preserve_state
Incremental Preserve internal data between runs.
-T --timing
TimeMe. Force display of timing information at end of process‐
ing.
-c --config=FILE
Use configuration file FILE
-n NAME
HostName. Use the hostname NAME.
-o --output=DIR
OutputDir. Use output directory DIR.
-t NAME
ReportTitle. Use NAME for report title.
F --logtype=TYPE
LogType. Specify log type to be processed. Value can be one of:
auto, clf, combined, domino, ftp or squid format. If not speci‐
fied, will default to auto format. FTP logs must be in standard
wu-ftpd xferlog format. A value of ‘auto’ states that the log
format automatically ascertained.
-f --fold
FoldSeqErr. Fold out of sequence log records back into analysis,
by treating as if they were the same date/time as the last good
record. Normally, out of sequence log records are simply ig‐
nored.
-Y CountryGraph. Suppress country graph.
-G HourlyGraph. Suppress hourly graph.
-x NAME
HTMLExtension. Defines the HTML file extension to use on the
created report files. If not specified, defaults to html. Do not
include the leading period.
-H HourlyStats. Suppress hourly statistics.
-L GraphLegend. Suppress color coded graph legends.
-l NUM GraphLines. Use background lines. The number of lines and where
to place are automatically calculated. For backwards compatibil‐
ity, any number > 0 enables. Use zero ('0') to disable the
lines.
-P NAME"
PageType. Specify file extensions that are considered pages.
Sometimes referred to as pageviews.
-m NUM VisitTimeout. Specify the Visit timeout period. Specified in
number of seconds. Default is 1800 seconds (30 minutes). Some‐
times referred to as sessions.
-I NAME
IndexAlias. Use the filename name as an additional alias for in‐
dex.
-M NUM MangleAgents. Mangle user agent names according to the mangle
level specified by num.
Mangle levels are:
5 - Browser name and major version
4 - Browser name, major and minor version
3 - Browser name, major version, minor version to two decimal
places
2 - Browser name, major and minor versions and sub-version
1 - Browser name, version and machine type if possible
0 - All information (left unchanged).
-g NUM GroupDomains. Automatically group sites by domain. The grouping
level specified by num can be thought of as 'the number of dots'
to display in the grouping. The default value of 0 disables any
domain grouping.
Hide Options
-a NAME
HideAgent. Hide user agents matching name.
-r NAME
HideReferrer. Hide referrer matching name.
-s NAME
HideSite. Hide site matching name.
-X NAME
HideAllSites. Hide all individual sites (only display groups).
-u NAME
HideURL. Hide URL matching name.
Table size options
-A --top_agents=NUM
TopAgents. Display the top num user agents table.
-R --top_refers=NUM
TopReferrers. Display the top num referrers table.
-S --top_sites=NUM
TopSites. Display the top num sites table.
-U --top_urls=NUM
TopURLs. Display the top num URL's table.
-C --top_countries=NUM
TopCountries. Display the top num countries table.
-e --top_entry=NUM
TopEntry. Display the top num entry pages table.
-E --top_exit=NUM
TopExit. Display the top num exit pages table.
Other Options
--use_geoip
Enables the use of the Maxmind GeoIP capability for more accu‐
rate detection of countries.
NOTE! Do not enable GeoIP if you analyse files that have had the
IP Address translated to a Fully Qualified Host Name. Use ei‐
ther raw IP Addresses and GeoIP, or Names and disable GeoIP.
ie. Don't use GeoIP AND DNShistory.
--match_counts
Display the various Group/Hide etc Match Counts. This option is
ideal for optimisation of the awffull.conf file. Just be careful
with optimising Agents in particular, as the order is typically
important.
CONFIGURATION FILES
See the awffull.conf(5) man page for complete details of all configura‐
tion options.
Configuration files are standard ASCII(7) text files that may be creat‐
ed or edited using any standard editor.
Blank lines and lines that begin with a pound sign ('#') are ignored.
Any other lines are considered to be configuration lines, and have the
form ‘Keyword Value’, where the ‘Keyword’ is one of the currently
available configuration keywords (see awffull.conf(5)), and ‘Value’ is
the value to assign to that particular option.
Any text found after the keyword up to the end of the line is consid‐
ered the keyword's value, so you should not include anything after the
actual value on the line that is not actually part of the value being
assigned. The file sample.conf provided with the distribution contains
lots of useful documentation and examples as well.
Certain "Keywords" will accept a 2nd value. In those situations, the
first value may be enclosed in double quotes (") to allow for white‐
space.
SEE ALSOawffull.conf(5)BUGS
None currently known. YMMV....
Report bugs to ⟨https://bugs.launchpad.net/awffull⟩, or use the email
discussion list: <awffull@stedee.id.au>
NOTES
In case it is not obvious: AWFFull is a play/pun on the word ‘awful’,
and is pronounced the same way. Yes it was deliberate.
REFERENCES
[1] Web Site Measurement Hacks. Eric T. Peterson (and others). O'Reil‐
ly. ISBN 0-596-00988-7.
2008-Dec-13 awffull(1)
