Custom analysis of web server logs

Web sites are intended to provide value for some set of users. One way to get feedback about their actual value is to analyze usage patterns of the site. Usage patterns can be found in the log files maintained by the server. While there are many capable programs available that analyze the logs, for some sites, the results from these programs are of minimal utility. This can occur because of the design of specific pages and their usage in the context of the entire site.

Using NOAA's SATPROD web site as an example, criteria are discussed for deciding on the need for a custom program. Creating a custom analysis program can be relatively simple. A programming language strong in text processing ability is desirable, like Python or perl, preferably one with rich intrinsic data structures. Examples are given of processing with Python.