Notice of emergency maintenance work scheduled for 30/07/2009 23:00 - 31/07/2009 01:00
NewNet engineers will be undertaking emergency maintenance to re-balance our BT centrals into the IPStream network tonight from 23:00.
Customers with DSL services via IPStream will see their sessions drop.
Most routers will reconnect automatically within 10 minutes; please reboot your router if it does not.
The Powerhost shared MySQL database server ‘mysql1′ experienced issues for around 40 minutes this afternoon due to high load. Customers using a MySQL database on their website may have noticed a decrease in performance or error loading their website.
This problem was quickly rectified by engineers. Maintenance is to be scheduled next week in order to pre-empt further issues and extra monitoring has been added to this server.
PowerHost server mysql1 has suffered high loading and errors this evening, causing some databases to become unavailable.
NewNet have just experienced a problem with a small number of routes to the internet in particular routes to google. This problem has now been resolved.
Reason for outage:
Google has experienced a problem with their peering sessions on the London Internet Exchange (LINX), where the sessions have gone down for around 30 minutes.
During that time, the linx reflectors where still propagating google routes even though those routes were not valid. This was caused by a problem with the firmware in these reflectors.
As a consequence NewNet (and other networks) routers where still seeing and preferring those routes to google (the ones the Linx reflectors where propagating and that were invalid) leading to a dead path and causing loss of connectivity to google.
NewNet engineers investigated the problem and have now shutdown all sessions to the Linx reflectors until these inconsistencies have been fixed by Linx engineers.
NewNet traffic is now taking an alternative path via a different peering exchange to google.
NewNet apologises for any inconvienced caused.
Resolved 14:00
Some customers using the post.newnet.co.uk cluster may have been experiencing slow mail deliveries over the past couple of hours. Engineers have resolved this issue and mail is flowing as usual. The service is being closely monitored.
