[CLOSED] BT WHOLESALE SYSTEM FALIURES

At 8:50am this morning we have recieved the following from BTW.....

BT WHOLESALE - MULTIPLE WHOLESALE SYSTEMS

We are currently experiencing a Major Service Failure

impacting a number of BT Wholesale Applications, including our
telephony System.

This means that we are unable to take calls from customers,
and provide updates on Orders, Faults or System Issues.

This is being investigated at the highest priority, and work
has commenced to restore service.

BT Wholesale regrets any inconvenience this may cause you and
will keep you updated within 2 hours.

------------------------------------------------------------------

[16:00] 25 July 2011

As in our previous mails, BT Wholesale have suffered a Major Service Failure today impacting a number of BT Wholesale Applications.

Our technical support teams have successfully restored the impacted hardware, which is allowing BT Wholesale and our Supplier to work on a full recovery of service to our customers.

BT Wholesale regrets the inconvenience this may cause you and would like to assure customers that we regard the extended unavailability of Wholesale Systems as a high priority.

A further update will be provided when we have more information regarding levels of service currently available.

Your continued patience during this incident is appreciated.

------------------------------------------------------------------

BT confirm issues resolved at 10pm

[CLOSED] BT link dropped

At 00:27 all traffic on our BT link stopped, and BGP timed out. This

is after some blip affectiong lots of lines at around midnight.
The results was all sessions dropping shortly after. The connection

resumed at 00:29



This looks the same as a few nights ago, which turned out to be

a planned work that had been sneaked in.

[CLOSED] - ADSL RADIUS Issues

We are currently investigating issues with our RADIUS servers, this may prevent some users authenticating whilst trying to connect.
We appreciate your patience whilst we resolve this issue.


Kind regards,


Support.


--------------------------------------------------------------------------



The RADIUS server issue has been resolved by our engineers, reconnection will be intermittent for the next 15 minutes while the servers are under load.


21CN customers may have to power down their equipment for 20 minutes to allow stale sessions to be disposed off.


We are awaiting an update from our engineers as to the cause of the outage.


Please accept our apologies for any inconvenience caused during this outage.


--------------------------------------------------------------------------


Our engineers will compile a report of the outage.

[CLOSED] - 20cn and 21cn outage

BT Wholesale appear to having a major outage, we’ve just seen all connections on the 20CN Reading RAS’, and 21CN Slough RAS’ drop – this is where BT’s network connects back to, rather than dependant upon the location your ADSL service is provided to.

This is a major BT outage affecting large parts of the UK, we’re awaiting an update from them at this time.

--------------------------------------------------------

BT have confirmed that engineers are on site and are still investigating the current DSL Outage.


We are awaiting further updates



--------------------------------------------------------


BT Wholesale have restored service to most customers during the night, however there maybe some intermittent results which are still under investigation.


Some customers may need to powercycle their equipment to restore service.
Any customers receiving the BT holding page will need to powerdown their router/modem for 20 minutes to clear any stale sessions.


--------------------------------------------------------


We are still awaiting the reason for this outage, once received this post will be updated.

[CLOSED] - Co-location/Broadband Outage

Dear Customer,

We have just experienced an outage affecting some Co location servers and ADSL customers, the initial findings are a PDU in Telehouse London had powered down due to electrical over current issues, as a result a full row of racks lost power.

We are awaiting confirmation of this however we have moved our equipment to another power phase.

Please accept our apologies for any inconvenience caused.

--------------------------------------------------


Following todays PDU outage one of our core switches has become unstable, as a precaution we will be replacing this device, this could result in intermittent or loss of connection for some customers till the work is complete.

We have a completion time of 3am (3rd March 2011).

Please accept our apology for the short notification as this is unavoidable emergency maintenance.

--------------------------------------------------

The core switch has been replaced, tested and is now in service.