We moved some core connection onto a new switch last night, since then we have seen much better performance, we will continue to monitor the situation.
Posted Dec 29, 2017 - 12:01 GMT
Identified
We have seen a recurrence of the packet loss issue and are upgrading the impacted core switch **NOW** to restore full service as quickly as we can.
This will momentarily cause disruption for approximately 50% of Wessex Internet customers who are downstream of this device for approximately 1-4 minutes.
Connectivity will restore automatically following this work.
We had attempted to wait until after our network freeze and plan this overnight, however as this issue is causing unfortunate repeated disruption we need to do this ASAP.
We apologise for any inconvenience this will cause.
Posted Dec 28, 2017 - 14:17 GMT
Monitoring
We were aware of widespread network degradation that manifested across the Wessex Internet Network from around 22:05 until 23:15.
We restarted a core switch at 23:15 which was suspected to be the root cause following diagnostic works, and this has resolved the problem.
This is a repeat of the same problem that occurred at 14th December.
This would have affected around 50% of the customers on the network.
Additional monitoring capabilities will be set up to alert us earlier about this type of issue if it repeats, which we hope it won't.
We shall also plan to take any mitigating steps tomorrow if appropriate to ensure that this does not repeat. If this takes place, additional notifications will be distributed.
We will be monitoring the network following tonight's incident.
Apologies for any inconvenience this outage caused.
(p.s. As a reminder to customers during these types of incidents, please DO NOT reset your routers as this will factory default your router to the manufacturer's settings, and not Wessex Internet's. A warning sticker is usually placed near this button advising as such. This will cause you to lose all access to the Internet until our engineers can remotely or worst case on-site restore your settings; this can only happen during business hours. Service restores automatically in times of network issues like this. You are however able to power off and on again (power cycle) any equipment however as part of your diagnostic steps.)