PDA

View Full Version : Installing AWStats (Linux)


Teej
11-05-2006, 07:55 AM
Download, check and extract the latest AWStats package (http://sourceforge.net/projects/awstats/):
cd ~/downloads
wget http://kent.dl.sourceforge.net/sourceforge/awstats/awstats-6.5.tar.gz
tar -tf awstats-6.5.tar.gz
tar -xzvf awstats-6.5.tar.gz -C /opt/
Note that I install all optional packages in /opt/ rather than allowing them to spread all over the system. I then create symbolic links in the locations where other packages might expect them. This way I can easily see all the add-ons installed whilst not having to mess about with the configuration of other packages that rely on them being in a specific location.

I also create version-independent symbolic links so I can upgrade and/or run multiple versions of the same package in parallel.
ln -s /opt/awstats-6.5/ /opt/awstats
ln -s /opt/awstats-6.5/ /usr/local/awstats
We can't usefully run the AWStats configuration script (/opt/awstats/tools/awstats_configure.pl) so we must manually customise the configuration files to reflect the Plesk configuration differences from a standard Apache httpd 2.0 configuration.

Plesk modifies the httpd.conf sequence to deal with its own per-domain, sub-domain, and the user's additional requirements. The sequence is:

/etc/init.d/httpd
/etc/sysconfig/httpd
/etc/httpd/conf/httpd.conf
/etc/httpd/conf.d/*.conf (on my VPS: zz0101_psa_httpd.conf)
/var/www/vhosts/<domain01>/conf/httpd.include
/var/www/vhosts/<domain01>/conf/vhost.conf
/var/www/vhosts/<domain01>/subdomains/<subdomain01>/conf/httpd.include
/var/www/vhosts/<domain01>/subdomains/<subdomain01>/conf/vhost.conf
/var/www/vhosts/<domain01>/subdomains/<subdomain02>/conf/httpd.include
/var/www/vhosts/<domain01>/subdomains/<subdomain02>/conf/vhost.conf
/var/www/vhosts/<domain02>/conf/httpd.include
/var/www/vhosts/<domain02>/conf/vhost.conf
/var/www/vhosts/<domain03>/conf/httpd.include
/var/www/vhosts/<domain03>/conf/vhost.conf
You should never edit each domain's httpd.include file because Plesk writes the contents based on what it has in its database - your changes are likely to be destroyed or cause problems for Plesk/httpd. Instead, create a vhost.conf in the same directory (/var/www/vhosts/<domain>/conf/vhost.conf) with the customised settings.

If it doesn't already exist create the vhost.conf file (make sure to do this as user root). Add the AWStats directives:
# Directives to add to your Apache conf file to allow use of AWStats as a CGI.
# Note that path "/usr/local/awstats/" must reflect your AWStats Installation path.
#
Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"
Alias /awstatscss "/usr/local/awstats/wwwroot/css/"
Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"
ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"
#
# This is to permit URL access to scripts/files in AWStats directory.
#
<Directory "/usr/local/awstats/wwwroot">
Options None
AllowOverride None
Order allow,deny
Allow from all
</Directory>
Note: AWStats requires the LogFormat be combined. Plesk, however, sets the log-file format to plesklog.

This isn't too much of a problem since the formats are almost always identical. combined is defined in /etc/httpd/conf/httpd.conf as:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
In /etc/httpd/conf.d/zz010_psa_httpd.conf Plesk defines plesklog depending on whether the module mod_logio is loaded, as:
<IfModule mod_logio.c>
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" plesklog
</IfModule>
<IfModule !mod_logio.c>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" plesklog
</IfModule>
The only difference being the %O in place of %b, which is the number of bytes sent in the HTTP Response. %O counts the Header bytes, %b doesn't. (see: Apache 2.0 Custom Log Formats (http://httpd.apache.org/docs/2.0/mod/mod_log_config.html#formats))

Remember: To have the vhost.conf for a domain/subdomain included in the start-up sequence you must issue this command to have Plesk add it to the sequence (adds an Include directive to the end of the VirtualHost in httpd.include):
/usr/local/psa/admin/sbin/websrvmng --reconfigure-vhost --vhost-name=<domain_name>
(See: Plesk: Customizable httpd.include per domain (http://download1.sw-soft.com/Plesk/Plesk7.1/Doc/html/plesk-7-reloaded-admin/apas02.html))

Now the configuration is complete the configuration files should be reloaded:
/sbin/service httpd reload
Reloading httpd: [ OK ]
Next, the awstats.<domain>.conf file needs creating. The template is /opt/awstats/wwwroot/cgi-bin/awstats.model.conf.

There is a lot of commentary in the model file which we don't really need in the production system so I've removed it in the interests of clarity (Reduces from 59,600 to 5,900 bytes) and use awstats.compact.conf as my standard domain template.
cat /opt/awstats/wwwroot/cgi-bin/awstats.model.conf | grep -v ^# | grep -v ^$ > /opt/awstats/wwwroot/cgi-bin/awstats.compact.conf
Hint: To remove comments and empty lines from any similar configuration file use this command: cat filename | grep -v ^# | grep -v ^$ > filename.compact

We need to create a directory for the per-domain configuration files and another for the AWStats data files:
mkdir /etc/awstats
mkdir /var/lib/awstats

In the next post I'll describe the per-domain settings.

Teej
11-05-2006, 09:22 AM
In this post I'm describing how to customise the per-domain settings of AWStats.

First, create the configuration file for our domain:
cp /opt/awstats/wwwroot/cgi-bin/awstats.compact.conf /etc/awstats/awstats.<domain>.conf
Now you can edit /etc/awstats/awstats.<domain>.conf using your preferred text editor. Much of the file doesn't require editing so I'll only show here the topmost entries where the settings that need changing for every domain reside.
LogFile="/var/www/vhosts/<domain>/statistics/logs/access_log"
LogType=W
LogFormat=1
LogSeparator=" "
SiteDomain="<domain>"
HostAliases="REGEX[<domain>.(net|org|com|co.uk|org.uk)] REGEX[www.<domain>.(net|org|com|.co.uk|org.uk)]"
DNSLookup=1
DirData="/var/lib/awstats"
DirCgi="/awstats"
DirIcons="/awstatsicons"
AllowToUpdateStatsFromBrowser=0
AllowFullYearView=2
EnableLockForUpdate=0
DNSStaticCacheFile="dnscache.txt"
DNSLastUpdateCacheFile="dnscachelastupdate.txt"
SkipDNSLookupFor=""
AllowAccessFromWebToAuthenticatedUsersOnly=0
AllowAccessFromWebToFollowingAuthenticatedUsers=""
AllowAccessFromWebToFollowingIPAddresses=""
CreateDirDataIfNotExists=0
BuildHistoryFormat=text
BuildReportFormat=html
SaveDatabaseFilesWithPermissionsForEveryone=0
PurgeLogFile=0
ArchiveLogRecords=0
KeepBackupOfHistoricFiles=0
DefaultFile="index.php index.jsp index.py index.html default.html default.htm"
SkipHosts="REGEX[^a.b.c.d]"

LogFile
Path to your domain/sub-domain's access_log file.

SiteDomain
The primary domain name of your site.

HostAliases
Any other hostnames the site answers to. To save listing many almost-identical entries you can use Regular Expressions as I've done here.

DNSLookup
Although it slows AWStats down, I prefer to have the hostnames of each IP address resolved (set to 1) because when viewing the reports it adds some context to what I'm seeing. By default this is set to 2.

DefaultFile
Its possible your site might use a variety of default files when a visitor accesses a directory without any filename specified. This is the list of possible filenames that covers PHP, Python, Java Server Pages, and the Microsoft standard default file.

SkipHosts
Especially during testing you probably don't want your own connections to the server to distort the statistics. This allows you to define a set of IPs that are ignored.

Once you've completed your changes its time to test it by running the awstats.pl PERL script to generate a database.
$ cd /opt/awstats
$ perl /opt/awstats/wwwroot/cgi-bin/awstats.pl -config=<domain> -update
Create/Update database for config "/etc/awstats/awstats.<domain>.org.conf" by AWStats version 6.6 (build 1.885)
From data in log file "/var/www/vhosts/<domain>/statistics/logs/access_log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 135
Found 0 dropped records,
Found 0 corrupted records,
Found 0 old records,
Found 135 new qualified records.
Now we need to generate a set of static XHTML pages containing the report:

Note: When you've decided where to write your report ensure the directories leading to that location have full permissions (777).
$ perl tools/awstats_buildstaticpages.pl -config=<domain> -awstatsprog=wwwroot/cgi-bin/awstats.pl -dir=/var/www/vhosts/<domain>/httpdocs/admin/reports/
Build main page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output
Build alldomains page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=alldomains
Build allhosts page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=allhosts
Build lasthosts page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=lasthosts
Build unknownip page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=unknownip
Build allrobots page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=allrobots
Build lastrobots page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=lastrobots
Build session page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=session
Build urldetail page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=urldetail
Build urlentry page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=urlentry
Build urlexit page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=urlexit
Build osdetail page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=osdetail
Build unknownos page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=unknownos
Build browserdetail page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=browserdetail
Build unknownbrowser page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=unknownbrowser
Build refererse page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=refererse
Build refererpages page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=refererpages
Build keyphrases page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=keyphrases
Build keywords page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=keywords
Build errors404 page: "wwwroot/cgi-bin/awstats.pl" -config=<domain> -staticlinks -staticlinksext=html -output=errors404
20 files built.
Main HTML page is 'awstats.<domain>.html'.

You'll notice that I have the output files written to the web-readable directory /admin/reports/ (which I have protected using Directory Authentication rules). Here's how to prepare and protect the location for a user called admin.

Edit /var/www/vhosts/<domain>/conf/vhost.conf and add the following section, adjusted for your domain:
<Directory "/var/www/vhosts/<domain>/httpdocs/admin">
AuthName "Administration"
AuthType Basic
AuthUserFile /var/www/vhosts/<domain>/conf/.htpasswd
Require valid-user
</Directory>
Now create the directories, the password file, the admin user, secure the password file, and reload the web-server configuration:
$ mkdir /var/www/vhosts/<domain>/httpdocs/admin
$ mkdir /var/www/vhosts/<domain>/httpdocs/admin/reports
$ /usr/bin/htpasswd -c /var/www/vhosts/<domain>/conf/.htpasswd admin
New password:
Re-type new password:
Adding password for user admin
$ chown apache.psaserv /var/www/vhosts/<domain>/conf/.htpasswd
$ chmod 640 /var/www/vhosts/<domain>/conf/.htpasswd
$ /sbin/service httpd reload
You can now view the report at http://<domain>/admin/reports/awstats.<domain>.html after authentication as admin.

When the .htpasswd file exists do not use the -c option when adding additional users - that will create a new, empty, .htpasswd file!
$ /usr/bin/htpasswd /var/www/vhosts/<domain>/conf/.htpasswd anotherUser
New password:
Re-type new password:
Adding password for user anotherUser
In my next post I'll show how to add awstats as a scheduled job using cron.

Teej
11-05-2006, 01:18 PM
In this post I'm describing how to have awstats run daily on all domains for which you've previously configured /etc/awstats/awstats.<domain>.conf files.

I've configured it so it runs before Plesk's preconfigured webalizer and the logrotate script.

Here's a summary of what the script does:

creates a list of the domains in the /var/www/vhosts/ folder
for each domain, checks for an awstats.<domain>.conf
if the web /admin/report/ directory doesn't exist, creates it
if the domain's access_log size is greater than 0, updates the statistics
generates a set of static pages for the report

Additional work is needed to have it handle sub-domains - I'll do this later.

Create the /etc/cron.daily/000awstats script file to be executed:
#! /bin/bash
# update AWStats statistics for the web site

# create a list of domains
DOMAINS=`ls -ld /var/www/vhosts/[!.]*[.]* | awk -F"/" '{print $5 }'`
# iterate the list
for DOMAIN in $DOMAINS
do
# ensure there's an awstats configuration for this domain
if [ -e /etc/awstats/awstats.$DOMAIN.conf ] ; then
# if the reports directory doesn't exist, create it
if [ ! -e /var/www/vhosts/$DOMAIN/httpdocs/admin/reports ] ; then
mkdir /var/www/vhosts/$DOMAIN/httpdocs/admin
mkdir /var/www/vhosts/$DOMAIN/httpdocs/admin/reports
fi
# update the statistics if the log file is not zero-sized
if [ -s /var/www/vhosts/$DOMAIN/statistics/logs/access_log ] ; then
perl /opt/awstats/wwwroot/cgi-bin/awstats.pl -update -config=$DOMAIN
fi
# call awstats to generate the static report pages
perl /opt/awstats/tools/awstats_buildstaticpages.pl -config=$DOMAIN -awstatsprog=/opt/awstats/wwroot/cgi-bin/awstats.pl -dir=/var/www/vhosts/$DOMAIN/httpdocs/admin/reports/
fi
done

exit 0
The filename prefix 000 is used to ensure awstats runs before other daily cron jobs - the jobs are executed in filename order.
Now change the file permissions so it can be executed:
$ chmod 755 /etc/cron.daily/000awstats

Teej
11-05-2006, 07:47 PM
Reports are often easier to understand when you can see the country (and sometimes city) that a visitor comes from.

AWStats provides loadable plug-ins to support the MaxMind GeoIP Country and City databases (http://www.maxmind.com/app/ip-location). Here's how to install and use them.

If you're not a subscriber to the MaxMind GeoIP databases you can get the lite versions of the Country (http://www.maxmind.com/app/geoip_country) and City (http://www.maxmind.com/app/geolitecity) databases.

Download and install the country and city databases (these are the URLs for the lite versions - if you have the full products alter this accordingly):
$ mkdir /opt/GeoIP
$ ln -s /opt/GeoIP /usr/local/share/GeoIP
$ cd /opt/GeoIP
$ wget http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz
$ wget http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
$ gunzip GeoIP.dat.gz
$ gunzip GeoLiteCity.dat.gz
$ ls
GeoIP.dat GeoIPCity.dat

Install the GeoIP Perl (http://www.maxmind.com/app/perl) module. To avoid having to install the C libraries I'll show how to install the PurePerl version.

Download, check, and extract the PurePerl GeoIP API:
$ cd ~/downloads
$ wget http://www.maxmind.com/download/geoip/api/pureperl/Geo-IP-PurePerl-1.17.tar.gz
$ tar -tf Geo-IP-PurePerl-1.17.tar.gz
$ tar -xzvf Geo-IP-PurePerl-1.17.tar.gz -C /opt/
Now install the package:
$ cd /opt/Geo-IP-PurePerl-1.17
$ perl Makefile.PL
$ make
$ make test
$ make install
Now you simply need to add additional lines to the /etc/awstats/awstats.<domain>.conf files(s). I've shown them here together with the tooltips plug-in. You might want to add these to the template config-file (/opt/awstats/wwwroot/cgi-bin/awstats.compact.conf)
LoadPlugin="tooltips"
LoadPlugin="geoip GEOIP_STANDARD /opt/GeoIP/GeoIP.dat"
LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /opt/GeoIP/GeoIPCity.dat"
Now when AWStats runs it'll add country and city details to the host details, as well as indicating the countries of origin.

epb
11-06-2006, 03:58 AM
Nifty, Ill be sure to look at this should I ever decide to install AWStats.