Standard Operational Procedures
Servers are monitored every 3 minutes from multiple external locations, using a third party service. In the event of a server not responding email and SMS alerts are sent to the support team. Server event logs are constantly monitored with Error, Alert and Information events being immediately sent to the support team for diagnosis. Any issues identified are generally resolved before they become a major problem.
Databases and applications from each server are backed up daily to network attached storage on a minimum 14 day cycle (90+ days for the severs at AWS and Azure). The log from each backup is emailed to the server team to ensure that backup windows are monitored.Backups are either to network attached devices inside the data centre or to Amazon S3 storage. The Amazon option gives the user the ability to independently download the backups (this is a read-only service and is protected by a User/Pass combination).
We use a third party service to take weekly (and in some cases daily) snapshots of the servers so that we can then start up a “clone” server if there is a disaster. This only applies to the servers at AWS, Azure and NextDC.
We use two third party services to monitor the servers: one from a physical infrastructure perspective and one from an end user perspective. The physical infrastructure reporting is dynamic and can be made visible to each client.
All servers are protected by a Ripe Group managed firewall/network security policy. Servers behind the firewall are on RFC1918 compliant private networks with no access from one subnet to another except via the firewall. The firewall also performs Intrusion Detection allowing us to identify probe attempts on the servers and ban suspect IP addresses for a period of 4 hours. This helps to protect the servers from organised attacks.
- Scheduled Maintenance:
We apply Microsoft Updates on the Friday following the second Tuesday of each month after consultation with each server owner. In the event of critical patches we attempt to apply these patches as soon as possible after consultation with the server owner. Linux patches are applied Monday mornings and the servers are restarted at 3am Tuesday morning (as agreed with each client). In the event of server compromise we take whatever remedial action is jointly considered appropriate. This may include isolation of the server from the network while this is carried out.