Daily Tasks

=This is a paste still need updating=

These are typical operational tasks of Deflect. See the Introduction to get started.

Edge management

Add a new edge

  1. Purchase a raw Debian amd64 7 image with ssh virtual private server ($vpsname), set root password

  2. Upgrade host

    root login; apt-get update ; apt-get dist-upgrade ; reboot
    
  3. Deploy Apache Traffic Server (ATS) and other prerequisites:

    controller# deploy.sh -A dnetname -b -H datacentreN

Note: deploy.sh uses a failsafe mechanism to minimize the risk of remote lock-out. Before applying firewall rules, the script creates a flag file on disk on the target node. After the remote script runs, it waits 30 seconds then checks for the file. If the file still exists, all rules are flushed. During those 30 seconds, {{DEFLECT_CONTROLLER}} waits a few seconds (for rules to be applied) then spawns a new ssh connection which deletes the flag. Therefore, if the rules are good, {{DEFLECT_CONTROLLER}} will have removed the flag by the time the edge looks for it, thus the edge should not ever lock us out.

  1. Nagios host configuration
    • scripts/nagios_config.sh updates Nagios configuration.
  2. Log collection
  3. Check ATS function
  4. Enable VPS

Enable an edge

  1. Pre checks Confirm edge is being monitored. Confirm edge is serving good content.

    controller# edge_check.sh -qc 10
    
  2. controller# edgemanage -A $dnetname --enable $vpsname

  3. Generate templated zone files and push to DNS

Disable an edge

  1. Remove from live edge list

    controller# edgemanage -A $dnetname –disable $vpsname

  2. Generate templated zone files and push to DNS

  3. Optionally silence alerts in nagios

Decommission an edge

  1. controller# edgemanage -A $dnetname --remove $vpsname
  2. Run log_collect.sh, or wait for nightly log collection
  3. Remove from monitoring
  4. Power down, burn, terminate, etc
  5. Remove $vpsname name from the $dnetname zone file
  6. Generate templated zone files and push to DNS

Update an ATS plugin

  1. build the plugin using axps

  2. copy into /usr/local/deflect/trafficserver/modules

    controller# deploy.sh -p ats -H $vpsname

Move a site from one network to another

  1. Move the zone file from /usr/local/deflect/etc/zones/$sourcezone/ to /usr/local/deflect/etc/zones/$destzone/
    • Optionally copy the zone file in $sourcezone to $sourcezone/holding
  2. Add the host origin to /usr/local/deflect/etc/zones/$destzone/$destzone.domain.zone
  3. Copy both deny and remap lines in remap.config from /usr/local/deflect/edges/$sourcezone/usr/local/trafficserver/conf/remap.conf to /usr/local/deflect/edges/$destzone/usr/local/trafficserver/conf/remap.conf and redeploy your ATS configuration.
  4. Run controller# zonegen.sh -A $destzone
  5. Check controller# rndc reload & tail -f0 /var/log/daemon.log for the new zone being loaded

Move an edge from one DNET to another

  1. Take the edge out of rotation
    • For edgemanage, use edgemanage -A dnet --remove $vpsname
  2. Remove the edge from the edge configs at /usr/local/deflect/etc/edges/edges.$dnetname
  3. Remove the edge from the originaldnet zone file - ‘’‘unless’‘’ the original dnet is in a primary configured primary dnet (all edges are kept in this dnet to support Nagios)
  4. Add the edge to the newdnet zone file and regenerate zones.
  5. Run controller# deploy.sh -A newdnet -b -H edge
  6. If needed (unlikely) reconfigure Nagios for this edge.
  7. Add the machine to rotation: controller# edgemanage -A newdnet --add edge    -c "Moving edge from originaldnet to newdnet"

ENCfs management

Mount ENCfs partition after a reboot

  • mountats host.deflect.ca - wait a few seconds, any prompts displayed will be answered automatically and require no input on your part.

Migrate a network to use ENCfs for ATS

  • Use migrate_to_encfs.sh, making sure to set the NETWORK variable.

Check edges are serving traffic

controller# /usr/local/deflect/scripts/edge_check.sh -h </code> <-- script is well documented, check the inline help or run without args

Check an individual edge

controller# /usr/local/deflect/scripts/edge_check.sh -H $vpsname

Managing DNS

Add a new DNS domain to controller and DNS

  1. cd /etc/bind/ && bvi named.conf.local
  2. Copy an existing block
  3. Put the appropriately named file into /usr/local/deflect/etc/zones if it’s to be autogenerated, or directly into /var/named/ if it’s to be static - the “notify” line tells bind to inform DNS of changes. For static files in /var/named, remember to increment the SOA serial - use unix epoch time (‘date +%s’)
  4. Restart bind
  5. Check /var/log/daemon.log for domain
  6. Test locally using ‘host’, ‘dig’, or ‘nslookup’
  7. Purchase domain on DNS
  8. Set DEFLECT_CONTROLLER to primary
  9. Generate templated zone files and push to DNS

Edit templated DNS entry

Template files are found in /usr/local/deflect/etc/zones/*.zone

  1. Edit the zonefile

    controller# cd /usr/local/deflect/etc/zones/ controller# bvi domain.com.zone

  2. Generate templated zone files and push to DNS

Generate templated zone files and push to DNS

  1. Generate zonefiles. zonegen.sh will add SOA and apex records in /var/named — read the script source for full details.

    controller# zonegen.sh

  2. Push changes to upstream DNS provider, and view logs

    controller# rndc reload & tail -0f /var/log/daemon.log

  3. Confirm all authoritative nameservers for the changed domain are now reporting the updated answers.

Managing partners

Test with a partner

  1. Configure remap.conf for partner site
  2. Add origin to zone file, regenerate zones
  3. Push configuration to a sample edge
  4. Test an edge with pac

Test an edge with pac

function FindProxyForURL(url, host) {
  if (shExpMatch(url, "http://samplesite.com*") ||
    (shExpMatch(url, "http://www.samplesite.com*"))) {
       return "PROXY sampleedge.deflect.ca"
  } else {
    return "DIRECT"
  }
}
  1. Create a pac file pointing to the sample edge.
  2. Configure a test browser with pac file; eg proxy settings | automatic proxy configuration url
  3. Set up the browser or extension (eg. Live HTTP Headers for Firefox
  4. Load site and verify it is “via” the sample edge

Add a partner

  1. Add new DNS domain to DEFLECT_CONTROLLER and DNS using existing records - put the zonefile directly into /var/named
  2. Delegate to DNS. NB: this step should ; meanwhile....
  3. Add a record to source.network.tld - something like partner.com.source.network.tld
  4. Push DNS; confirm partner.com.source.network.tld resolves
  5. Editing remap.config, add the record to ATS config - use another partner as an example
  6. Confirm all well and good with old records on new DNS host
  7. Add partner to stats host
  8. Engage partner for testing by setting their browser proxy to edge.your.dom (or create a .pac file
  9. Strip out SOA and apex records (both A and NS, but not MX) to create zone template in /usr/local/deflect/etc/zones/holding. Replace non-apex A records with CNAME -> edge.your.dom where appropriate
  10. Confirm time with partner for go-live
  11. At agreed time, move zone template from zones/holding/ into zones/, Generate templated zone files and push to DNS(#Generate_templated_zone_files_and_push_to_DNS]

Apache Traffic Server configuration

Generally, you’ll only need to change remap.conf. records.conf is used for site specific configuration. Examine the files to see how they are configured, and consult Apache Traffic Server documentation.

  • Edit ATS config:

    controller# cd /usr/local/deflect/edges/your_DNET/usr/local/trafficserver/config bvi file_to_edit.config

  • This will keep a dated backup of your edit, and mails a diff to sysadmin for monitoring.

  • Push the file to a staging host- in this example your_edge.your.dom

    cd .. deploy.sh -A your_DNET -p ats -H your_edge

  • Configure your browser to use your_edge.your.dom as a proxy for sites, conduct manual testing;

    • The things you have changed
    • A few things you haven’t changed too, just in case
  • Deploy new configuration

    controller# deploy.sh -A your_DNET -p ats -H ALL

  • Disable proxies and test again, results should be same.

Fail2ban rules

  • Files are in /usr/local/deflect/etc/fail2ban/

  • Edit with bvi.

  • They are deployed using:

    controller# deploy.sh -A your_DNET -p fail2ban -H your_edge (or ALL)

Revert a config change

controller# ls -ltr old/borked_file.config*
  • check for the last revision before the bad edit, then

    controller# bvi borked_file.conf

  • Use the editor to read in the old file. For example, in vim:

    dG:r old/borked_file.conf-date-name ddZZ

Monitoring tasks

Access Nagios

https://deflect_controller/nagios3/index.php

Acknowledge an alarm

  • Log into the Nagios web interface
  • Select the status detail of the test which is alarmed
  • Select “acknowledge”

Change monitoring configuration

  1. Consult the nagios documentation e.g. Nagios Docs

  2. Make a change to the nagios config

    controller# cd /etc/nagios3/conf.d bvi file_to_edit.cfg

  3. Test the config

    controller# /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg

  4. Fix any problems encountered, and once tests pass, have nagios re-read its config

    controller# /etc/init.d/nagios3 reload

Add a host to monitoring

If you’ve built a host from one of the scripts, it will be ready and waiting to be monitored- the nrpe agent will be running, accepting connections from DEFLECT_CONTROLLER and any backup hosts.

  1. Go to nagios host dir

    controller# cd /etc/nagios3/conf.d/hosts

  2. Copy an existing host- for example another edge

    controller# cp host.rackspace1.cfg host.datacentreN.cfg

  3. Edit new file

    controller# bvi host.datacentreN.cfg

update hostname references 4. Test and reload as above.