Tuesday 11 January 2011

[CLOSED] - 21CN ADSL Network Outage

We are currently experiencing issues with the 21CN ADSL network, this is under investigation with BT.

This outage is affecting most ISP's.

Once we have an update we will inform you.

Thank you.

186k Support.

----------------------------------------------------------------


Update: 09:06

It has been discovered that "21CN-BRAS-RED2" dropped all lines at approx 04:30 this morning, this has had a knock on affect to other BRAS.

BT are currently working on BRAS-RED2,

We are awaiting further updates.

----------------------------------------------------------------

Update: 10:34

There is no longer an issues with any 21CN BRAS, however BT can see that authentication requests are not reaching ourselves.

BT are working on collapsing the tunnels to ourselves allowing these to re establish and authentication requests to reach our RADIUS.

We are expecting to see connections within the next 30 minutes.

----------------------------------------------------------------

BT are still working on the tunnels allowing authentication requests to reach us.

Please accept our apology for the delay.

----------------------------------------------------------------

It appears the resolution is taking longer than anticipated, we are awaiting an update from BT but will continue to chase this.

Once we have any more information we will inform you.

----------------------------------------------------------------

Update 12:25

Our network engineers have identified the issue within BT's network and are working with BT to resolve the fault, This has been identified as a device configuration error that was highlighted when the tunnels were collapsed, BT are looking at the configuration and will be looking to correct this.

Next update will be in 1 hour (13:25)

Please accept our apologies for the time it has currently taken to resolve this issue.

----------------------------------------------------------------

Update: 13:22

BT Operations are rebuilding the configuration on the device that controls the tunnels, once this configuration is rebuilt it will be loaded onto the device.

The rebuilding and reload will be complete within the next 30 minutes.

We will have an update in 30 minutes (13:52).

----------------------------------------------------------------

Update: 13:52

Configuration has now been reloaded, we now expect users to start connecting, however there will be an initial load on the RADIUS servers so connection could take longer than usual to initiate.

Any users that are receiving the BT holding page will be required to power down their equipment for at least 20 minutes and then attempt to reconnect.

We will monitor the connections and report how this is going in 1 hour (14:52)

Thank you for your patience.

----------------------------------------------------------------

We can see users are now connecting and believe the issue to be resolved.

BT are investigating the issue with the configuration file, once we have the explanation we will update this post (although the post will be closed).

Please accept our apologies for any inconveniences caused during this issue.

186k Support,