1 Log Analyzer: Integration of awstats with the Apache http server {anchor:Log Analyzer: Integration of awstats with the Apache http server} Lately I've been asking myself some questions like: 1. How many daily visits I'm getting? 1. From where does those visits come from? 1. Which pages are the most viewed? 1. ~~...~~ Well you know, the normal questions you ask yourself when you have a website. I was noticing a traffic increase while reading the adsense's reports but the information wasn't very useful I just saw I was getting more ads displayed.. That actually didn't tell me, for sure, that I was having more visitors. So I decided to install a log analyzer for apache. \\After some search I decided to go for {link:awstats|url=http://awstats.sourceforge.net|newWindow=true}. In this post you'll find an introduction to what awstats is and it's capabilities along on how to deploy it. Keep in mind that the solutions presented here are the ones I designed, which are probably similar to some out there and probably also very different from others. It's up to you to see if they fit in your needs or if you have to adapt them. The following subjects will be presented: * Before starting * Presenting awstats * Installing awstats * Configuring awstats * Configuring Virtual Hosts in Apache __Before Starting__ You need to have installed the following software: * {link:Apache HTTP Server|url=http://www.apache.org|newWindow=true}: other http servers will also work with awstats, but I'll be lecturing with apache since it's the one I use. * {link:Perl 5.00503 or higher|url=http://www.perl.org/|newWindow=true}: you'll probably already have perl installed with your linux distro, and it will be higher then 5.00503. * {link:Awstats|url=http://awstats.sourceforge.net|newWindow=true}: if I really convince you that awstats is a nice webstats tool and you want to install it. There's no special compilation options for Apache, since the directives we'll be using (<__VirtualHost__> and <__Directory__>) are part of the core system. __Presenting awstats__ Awstats is a perl script which reports your website statistics using the access log of the server. It also supports ftp and mail logs, although, that is out of the scope of this post. It can operate as a cgi-bin file within your http server or it can run in batch mode and generating a static HTML file. Awstats supports the following features (this is not an exhaustive list of the features. For such list please check {link: awstats comparison|url=http://awstats.sourceforge.net/docs/awstats_compare.html|newWindow=true} with other log analyzers): * Unique visitors, number of visits, number os pages retrieved and bandwidth spent counters by month and day of the month * Top day of the week, top hour of the day * Countries and hosts that have been accessing your site (with some plugins you can even show the city) * Spiders that have crawled your website * Visit Duration * Top pages requested * Operating Systems from the visitors * Browsers from the visitors * Referees * Search Terms __Installing awstats__ Installing it's pretty straight forward, you first download {link:awstats|url=http://awstats.sourceforge.net/#DOWNLOAD|newWindow=true}. You ~~untar~~ the file into your /usr/local directory and then you run ~~perl awstats_configure.pl~~ and answer the simple questions. When you finish this process you'll have a configuration file in __/etc/awstats/awstats.YOUR_HOST.conf___ It also probably added you a few lines in the http.conf or generated a file with the lines that it then asks you to add. For now we'll ignore those lines. At least you'll probably be interested in one plugin, called geoIPFree, which is a plugin that allows awstats to report from which country a certain IP is in a more accurate way. So you can {link: download GeoIP free|url=http://search.cpan.org/~gmpassos/Geo-IPfree-0.2/|newWindow=true} and install it. Now before we configure awstats, we'll first configure the virtual hosts in apache. __Configuring Virtual Hosts in Apache__ The idea is that we setup two different virtual hosts, one that holds your website and other one that displays the stats. This is actually not necessary, but I prefer this way. Imagine that your apache installation is located at __/usr/local/apache__, your log directory is __logs__ and your httpd documents directory is __htdocs__.\\ Your vhost configuration has to look something like this: {quote} NameVirtualHost * <VirtualHost *>\\ ServerName www.serverdomain.tld\\ DocumentRoot /usr/local/apache/htdocs/mainsite/\\ ErrorLog /usr/local/apache/logs/mainsite-errors.log\\ LogFormat "%h %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-agent}i\\\""\\ TransferLog /usr/local/apache/logs/mainsite-access.log\\ </VirtualHost>\\ <VirtualHost *>\\ ServerName awstats.serverdomain.tld\\ DocumentRoot /usr/local/apache/htdocs/stats/\\ ErrorLog /usr/local/apache/logs/stats-errors.log\\ TransferLog /usr/local/apache/logs/stats-access.log\\ </VirtualHost> {quote} Probably the only line that left you thinking is the LogFormat one, that type of format is called the NCSA format and it's a widely adopted standard. Using it you can have access to details such as referees, browser and operating system of the clients that are accessing your website. Although we might want to restrict access to our statics, so we'll configure the access policy for the awstats vhost, to do that you need to use the <__Directory__> directive. So inside the awstats VirtualHost we add the following: {quote} <Directory /usr/local/apache/htdocs/stats/>\\ Order Deny, Allow\\ Allow from 192.168.0.0/24\\ Deny from all\\ </Directory> {quote} With such configuration only local IPs in the private network 192.168.0.0/24 will be allowed to access the awstats reports. __Configuring awstats__ Open the configuration file located in __/etc/awstats__ and make sure that: 1. __LogFile__ is set to your mainsite TransferLog file 1. __LogType__ is set to W 1. __LogFormat__ is set to 1 (the NSCA format) 1. __SiteDomain__ is set to your site domain 1. __AllowToUpdateStatsFromBrowser__ is set to 0 (not that we will actual use it) 1. __DNSLookup__ is setted to 1 (this means that dns is fully enabled and every hostname will be resolved. If you have a high traffic site, this might take a while to process) 1. __LoadPlugin="geoipfree__ is uncommented Now we'll have to setup a cron job that will update the reports, you can choose daily or hourly. That's up to you. You now have to choose between using static HTML or having a cgi-bin script, the first one is more secure but less flexible, if you want for example to check the reports of daily visits for a previous month you'll have to generate by hand again a new report, using the cgi-bin you just have a drop box and have to select the month you want. Although awstats already had some security problems in the past and it's up to you to know which one to choose. Let's first configure for a static output. You write the following script {quote} #!/bin/bash perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=YOUR_DOMAIN -update -output -staticLinks > /usr/local/apache/htdocs/stats/index.html {quote} And you store it in the __cron.daily__ or __cron.hourly__ directory so it gets executed daily or hourly. Don't forget to give executing permissions to it. You can now access your website statistics reports by simply typing in your browser __awstats.yourdomain.tld__. And you're done! Now let's write the script to use with awstats in a cgi-bin way. {quote} #!/bin/bash perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=YOUR_DOMAIN -update {quote} Although that's not all, you still have to do two other things: \1. Create a Symlink from your /usr/local/apache/htdocs/stats to /usr/local/awstats/wwwroot 2. Add the following lines within the awstast vhost: {quote} Alias /awstatsclasses "/usr/local/apache/htdocs/stats/classes/"\\ Alias /awstatscss "/usr/local/apache/htdocs/stats/css/"\\ Alias /awstatsicons "/usr/local/apache/htdocs/stats/icon/"\\ ScriptAlias /cgi-bin/ "/usr/local/apache/htdocs/stats/cgi-bin/"\\ {quote} Now you can access your website statistics reports by going to the following URL: awstats.yourdomain.tld/cgi-bin/awstats.pl?config=The_Server_Name_You_Give_In_The_Config_Param __Final notes__ As you probably already realized it's pretty simple to scale this solution. you just need to get more awstats configuration files, and run each configuration file in the cron job. \\ If you use the cgi-bin version you just have to change the __config__ parameter to access each stats but if you are using the static HTML solution you have to generate a different HTML file for each one (which is trivial). I think there's not much else to add. I suggest you to read the Documentation in the awstats and also the comments in the configuration files. 2006-05-24 20:23:18.154 2006-05-24 20:23:18.154 Ok this is not really related to how useful awstats is or any other number/stats prodocuing software out there (we have awstats, webalizer and nagios running here nonstop) but at what point do you get number/information overload to where the information is no longer really usefeull because there is so much of it? I find my self in that position more often than not which is not really good considering I am a system adming and I can only imagine what a normal user who log into their webhopsting company and looks at their awstats and goes "WTF IS THIS, WHAT DOES IT ALL MEAN?! /slams head against desk..." 2006-05-25 17:35:07.936 2006-05-25 17:35:07.828 Well I don't know nagios, but I do know awstats and webalizer.\\ They both spit plenty of data, but I think it's well organized. Maybe awstats has too much things, but I guess that most of them are self explanatory and you can extract valuable information from some of the information, like for example: * If your visits are going up (always a good sign) * What's your most viewed pages * The 404 your server is issuing. But since I never had to analyze awstats for a server that has for example 10000 hits a day, I don't actually have the feel of doing such work. Although I do believe that in such cases it might be difficult to process data...Maybe if you keep you head clear, restrict the working set to just the top 10-20 lines of each table that display relevant info you might get along (or not). 2006-05-25 20:11:47.912 2006-05-25 20:11:47.825 Well i was just talking about information overload in general webstats was just a the simplest example... it seems like these days the goal of everyone is to get you information overload headache :) with having rss feeds, mailing lists, blogs, forums. There seems we have gotten into the age of Information Overload not the Information Age... but that is just my opinion. 2006-05-25 20:41:29.15 2006-05-25 20:41:29.086 Well it's not overload, it's redundancy! When an information link fails you still have plenty of others ~~laugh~~. For example I have subscribed plenty of rss feeds and mailing lists... Most of the times I just browse through the titles, maybe 1 in each 20 or even 30 calls my attention... So I don't actually get overloaded. And to be truth I never had that overload headache, maybe it's just me! 2006-05-25 23:13:35.765 2006-05-25 23:13:35.7 Well see thats what I am talking about. Why waste the resources/time to sift through 20/30 things and actually read 1... thats exactly the waste of time I am talking about :). 2006-05-26 17:19:57.461 2006-05-26 17:19:57.375 Well whe never know when such topic might get in handy in the future. Going through them __might__ ring a bell when the future I may need something related with that topic, or maybe it won't or I never will actually need it. But I don't actually see as a waste of resources/time as you say. It's information and it's always valuable. I see this mostly has a graph problem (I really do). Imagine the following: 1. Map all the information you have in a graph 1. Give weights to the edges acording to your interests in that day 1. Browse the graph in a way of maximizing your Interest Now if your interestes change, which they do... You can always go back to the graph or learn something else, even if you didn't even look at it the 1st time! The point here, is that reading the subjects helps me ~~index~~ my ideas, even if I don't read the message body. It's like reading the newspaper and only looking at the headlines. It's not a waste of time, it's actually a maximization of used time! ~~laugh~~ But that's my point of view, you're totally free to have your own! 2006-05-27 00:37:58.138 2006-05-27 00:37:58.034