Logging

Deflect ‘Cruncher’ Awstats Log Analysis

This is the method Deflect does its Awstats setup for all edges running Apache Traffic Server.

Directory Structure and Files

  • Installed in /var/deflect/
    • etc/
      • edges
      • id_rsa
    • bin/
      • 0_backup_awdata.sh
      • 1_log_collection.sh
      • 2_uz2live.sh
      • 3_resolve_logs.sh
      • 4_crunch_configs.sh
      • add_aw.sh
      • cron_crunch.sh
    • archives/
      • edge1.domain.com/
      • ... /
      • edge24.domain.com/
    • tlive/
      • 201306/
      • 201307/
      • ... /
      • 201401/
    • backup/
      • var_lib_awstats-20140103010001-341825787/
      • etc_awstats-20131201000138-262232287/
    • lock/

The Scripts ( bin/ )

  • 0_backup_awdata.sh
    • This script just does a copy of the config and awstats data files before running anything. For easier recovery if anything goes wrong.
  • 1_log_collection.sh
    • This script reads a list of edges and rsync the ATS log files ‘-.gz’ to a archive
  • 2_uz2live.sh
    • Gets date and looks for zipped file in archive that match the right date. Unzips to a temp location in a separated YYYYMM directory If file was already unzipped in this directory before it does a ‘cat /dev/null > old_file’
  • 3_resolve_logs.sh
    • Uses some of awstats tools to merge the unzipped log files together into one file YYYMM/current.log
  • 4_crunch_configs.sh
    • Read all the Awstats configs setup and forks the to all crunch using YYYYMM/current.log
  • cron_crunch.sh
    • Runs all the files above
  • add_aw.sh
    • A simple config creator script to add a client to Awstats
    • you need (domain, config name, password)
    • Also gives a option to manually add REGEX for advanced REGEX

Configuration ( etc/ )

  • edges
    • List of all edges we are collecting log files from.
  • id_rsa
    • Private ssl key so scripts can access (rsync) logs

Locking ( lock/ )

Scripts write files in here to make sure things finish. If a problem occurs, like a edges could not get collected from, you can later just delete the file inside this directory and do a manual run.