Wednesday 30 November 2011

[CLOSED] - BT System Issues

BT service was restored at 3am


Any outstanding issues will now be progressed with BT


Apologies for any delays cause by this issue


---------------------------------------------------
Unfortunatley BT have not fixed their issue and the new update time for a fix is 18:30


Apologies for any inconvenience caused
---------------------------------------------------


BT Have stated their internal systems are not expected to be operational before 16:45, however we should receive a further update at approx 14:00


---------------------------------------------------


BT are currently experiencing issues with their internal systems, this has temporarily stopped our ability to raise faults with BT.


The next update is expected at 1pm

Wednesday 23 November 2011

[CLOSED] - Intermittent Network Issues

22.50 - The network appears to have stabilised however investigations are still ongoing.


---------------------------------------------------------


Dear Customer,

We are currently experiencing intermittent network issues.

Our network engineers are investigating the cause of this issue.

Once we have an update we will inform you.

The 186k Team

Friday 4 November 2011

[CLOSED] - Intermittent Network Issues

16:50 - The network outage is confirmed as clear

The problem was caused by heavy load on a core device. This was not correctly reporting. After locating the issue a reboot of the device has fixed the problem. Future monitoring of this device has been amended.

Please accept our sincere aplogies for any inconvenience caused


----------------------------------------------------------

15:40 - The network appears to have stabilised however investigations are still ongoing.

Further updates shortly

----------------------------------------------------------


Dear Customer,

We are currently experiencing intermittent network issues.

Our network engineers are investigating the cause of this issue.

Once we have an update we will inform you.

The 186k Team.

Friday 21 October 2011

[CLOSED] All BT ADSL lines are down

BT ADSL lines are down
There is a BT outage in Stepeny Green where we (and others) interconnect in to BT.

BT have raised an Incident and are aware of a problem



Update: 9:15
Lines are coming back - most 20cn circuits are back online

For 21CN circuits please power off your router for at least 20 minutes then retest as you may have the BT termination IP address starting 172.*.*.*

Update:12:00
This incident is now closed.

The issue was also reported by ThinkBroadband

http://www.thinkbroadband.com/news/4829-major-broadband-outage.html

Friday 14 October 2011

[CLOSED] Sheffield Major Service Outage

BT currently have a major outage at Sheffield affecting a large surrounding area of South Yorkshire and the Midlands.

IMT32513/11 21CN CORE/METRO NODE : SHEFFIELD : LOSS OF SERVICE:Service affecting:Total Loss;

Estimated restore time: 14-10-2011 13:20:00


Update


Today 16:25:00 BT confirm that they are aware of the issue, and have re-opened the earilier incident they had.


Update


Today 16:52:00 There appears to be an issue with a card on the BRAS and BT are  currently sourcing a replacement

Estimated Completion Date: 14/10/2011 18:00:00


Update
Lines came back up at approx 8pm on Friday.

Wednesday 14 September 2011

[CLOSED] - Telephone Problems

Dear Customer,


We are currently experiencing intermittent issues with our phone system, some calls will be held within the queue but not answered.

During this issue please use the premier@186k.co.uk email address as the primary contact method.

Kind regards,
The 186k Team

Friday 26 August 2011

[CLOSED] - BT Wholesale 21cn Outage



BT are currently investigating an outage affecting 21cn circuits in and around the London area.


There was an network drop this morning (BRAS-RED1-L-FAR) and a number of lines failed to reconnect, this is being treated an a BT internal incident.


We are awaiting information from BT


Kind regards,


Support.




---------------



We are now seeing affected lines reconnecting, however we are awaiting a report from BT.


Some routers may need to be power cycled to aid reconnection.


Tuesday 9 August 2011

[CLOSED] BT 20CN CONNECTIVITY ISSUES

At 12:20 pm we recieved the following from BT.....

We are currently experiencing difficulties with 20 CN Broadband, that may cause a small portion of your customers on the IPStream product to experience a loss of PPP Connectivity


Work has commenced to restore service, with DLM being temporarily suspended to avoid any further disruption to End User Service.

A script is being run to repair corrupted records, then all impacted circuits will be forced back through DLM to restore service. At this stage normal DLM will also be put back in to service.

Wholesale Customer Service would like to ask that during this incident, customers do not contact BT Wholesale to report DLM related Issues.



BT Wholesale regrets any inconvenience this may cause you and will keep you updated within 2 hours.


[16:30] - update from BT

Further to our previous message, our technical teams are still working to fix the system problem on 20CN Broadband Systems.


A script is being run to repair corrupted records, then all impacted circuits will be forced back through DLM to restore service. At this stage normal DLM will also be put back in to service.

Wholesale Customer Service would like to ask that during this incident, customers do not contact BT Wholesale to report DLM related Issues.

BT Wholesale is working hard to resolve this issue to restore normal service to the application.

BT Wholesale regrets any inconvenience this may cause you and would like to assure customers that we regard the extended unavailability of 20CN Broadband Systems as a high priority and is investigating how we can restore service.

BT Wholesale will provide further updates within 2 hours.


12:40 10th August 2011

We are pleased to report that the DLM service has been


restored to 20CN Broadband Systems.



BT Wholesale regrets any inconvenience which may have been

caused by these issues.





Monday 25 July 2011

[CLOSED] BT WHOLESALE SYSTEM FALIURES

At 8:50am this morning we have recieved the following from BTW.....

BT WHOLESALE - MULTIPLE WHOLESALE SYSTEMS

We are currently experiencing a Major Service Failure

impacting a number of BT Wholesale Applications, including our
telephony System.

This means that we are unable to take calls from customers,
and provide updates on Orders, Faults or System Issues.

This is being investigated at the highest priority, and work
has commenced to restore service.

BT Wholesale regrets any inconvenience this may cause you and
will keep you updated within 2 hours.

------------------------------------------------------------------

[16:00] 25 July 2011

As in our previous mails, BT Wholesale have suffered a Major Service Failure today impacting a number of BT Wholesale Applications.

Our technical support teams have successfully restored the impacted hardware, which is allowing BT Wholesale and our Supplier to work on a full recovery of service to our customers.

BT Wholesale regrets the inconvenience this may cause you and would like to assure customers that we regard the extended unavailability of Wholesale Systems as a high priority.

A further update will be provided when we have more information regarding levels of service currently available.

Your continued patience during this incident is appreciated.

------------------------------------------------------------------

BT confirm issues resolved at 10pm

Thursday 19 May 2011

[CLOSED] BT link dropped

At 00:27 all traffic on our BT link stopped, and BGP timed out. This

is after some blip affectiong lots of lines at around midnight.
The results was all sessions dropping shortly after. The connection

resumed at 00:29



This looks the same as a few nights ago, which turned out to be

a planned work that had been sneaked in.

Wednesday 6 April 2011

[CLOSED] - ADSL RADIUS Issues

We are currently investigating issues with our RADIUS servers, this may prevent some users authenticating whilst trying to connect.
We appreciate your patience whilst we resolve this issue.


Kind regards,


Support.


--------------------------------------------------------------------------



The RADIUS server issue has been resolved by our engineers, reconnection will be intermittent for the next 15 minutes while the servers are under load.


21CN customers may have to power down their equipment for 20 minutes to allow stale sessions to be disposed off.


We are awaiting an update from our engineers as to the cause of the outage.


Please accept our apologies for any inconvenience caused during this outage.


--------------------------------------------------------------------------


Our engineers will compile a report of the outage.

Wednesday 30 March 2011

[CLOSED] - 20cn and 21cn outage

BT Wholesale appear to having a major outage, we’ve just seen all connections on the 20CN Reading RAS’, and 21CN Slough RAS’ drop – this is where BT’s network connects back to, rather than dependant upon the location your ADSL service is provided to.
This is a major BT outage affecting large parts of the UK, we’re awaiting an update from them at this time.

--------------------------------------------------------


BT have confirmed that engineers are on site and are still investigating the current DSL Outage.


We are awaiting further updates



--------------------------------------------------------


BT Wholesale have restored service to most customers during the night, however there maybe some intermittent results which are still under investigation.


Some customers may need to powercycle their equipment to restore service.
Any customers receiving the BT holding page will need to powerdown their router/modem for 20 minutes to clear any stale sessions.


--------------------------------------------------------


We are still awaiting the reason for this outage, once received this post will be updated.





Wednesday 2 March 2011

[CLOSED] - Co-location/Broadband Outage

Dear Customer,

We have just experienced an outage affecting some Co location servers and ADSL customers, the initial findings are a PDU in Telehouse London had powered down due to electrical over current issues, as a result a full row of racks lost power.

We are awaiting confirmation of this however we have moved our equipment to another power phase.

Please accept our apologies for any inconvenience caused.

--------------------------------------------------


Following todays PDU outage one of our core switches has become unstable, as a precaution we will be replacing this device, this could result in intermittent or loss of connection for some customers till the work is complete.

We have a completion time of 3am (3rd March 2011).

Please accept our apology for the short notification as this is unavoidable emergency maintenance.

--------------------------------------------------


The core switch has been replaced, tested and is now in service.


Wednesday 23 February 2011

[CLOSED] authentication Issues

We are currently experiencing issues with radius authentication.
Due to this 21CN Circuits may get the BT termination page and/or a BT IP address.

In this situation please ask your end user to power off their router/equipment for a minimum of 20 minutes and retest.

The issue is being investigated as top priority.

Apologies for any inconvenience caused. Further updates will be posted ASAP

------------------------------------------------------------------

The RADIUS authenticated issue is now resolved.

Once again please accept our apologies for any inconveniences caused during this outage.

The 186k Team

------------------------------------------------------------------

Monday 14 February 2011

[CLOSED] - LLU Outage

Apologies, Our LLU suppliers have stated....

We have lost a fibre connection between Telehouse North and London Hosting Centre causing a loss of some resilience, and forcing traffic to take other routes, which is causing some speed issues for a small number of customers.Our dark fibre providers are aware and investigating this issue – we hope to have normal service resumed shortly.

Further updates when recieved

----------------------------------------------------------------
[13:20] This issue has now been cleared by our LLU suppliers

Normal service has now been resumed

Apologies for any inconvenience

----------------------------------------------------------------

Thursday 10 February 2011

[CLOSED] - BT Openreach MBORC Declaration

BT Openreach have declared a “Matter Beyond Our Reasonable Control” (MBORC) in parts of the North of Scotland:Following lightning and very high winds in parts of the North of Scotland, Openreach is declaring Matters Beyond our Reasonable Control (MBORC) in the following areas with effect from 14.00 09 February 2011:Highlands and IslandsNorth East ScotlandThe declaration will be applied only to our repair activities and only to faults which have been caused by this incident of exceptionally severe weather.

Wednesday 2 February 2011

[CLOSED] - LLU Availability Checker

Our LLU supplier is currently experiencing issues with their availability checker, this is currently not returning results.

Engineers are investigating this.


-------------------------------

This has now been resolved.

Tuesday 11 January 2011

[CLOSED] - 21CN ADSL Network Outage

We are currently experiencing issues with the 21CN ADSL network, this is under investigation with BT.

This outage is affecting most ISP's.

Once we have an update we will inform you.

Thank you.

186k Support.

----------------------------------------------------------------


Update: 09:06

It has been discovered that "21CN-BRAS-RED2" dropped all lines at approx 04:30 this morning, this has had a knock on affect to other BRAS.

BT are currently working on BRAS-RED2,

We are awaiting further updates.

----------------------------------------------------------------

Update: 10:34

There is no longer an issues with any 21CN BRAS, however BT can see that authentication requests are not reaching ourselves.

BT are working on collapsing the tunnels to ourselves allowing these to re establish and authentication requests to reach our RADIUS.

We are expecting to see connections within the next 30 minutes.

----------------------------------------------------------------

BT are still working on the tunnels allowing authentication requests to reach us.

Please accept our apology for the delay.

----------------------------------------------------------------

It appears the resolution is taking longer than anticipated, we are awaiting an update from BT but will continue to chase this.

Once we have any more information we will inform you.

----------------------------------------------------------------

Update 12:25

Our network engineers have identified the issue within BT's network and are working with BT to resolve the fault, This has been identified as a device configuration error that was highlighted when the tunnels were collapsed, BT are looking at the configuration and will be looking to correct this.

Next update will be in 1 hour (13:25)

Please accept our apologies for the time it has currently taken to resolve this issue.

----------------------------------------------------------------

Update: 13:22

BT Operations are rebuilding the configuration on the device that controls the tunnels, once this configuration is rebuilt it will be loaded onto the device.

The rebuilding and reload will be complete within the next 30 minutes.

We will have an update in 30 minutes (13:52).

----------------------------------------------------------------

Update: 13:52

Configuration has now been reloaded, we now expect users to start connecting, however there will be an initial load on the RADIUS servers so connection could take longer than usual to initiate.

Any users that are receiving the BT holding page will be required to power down their equipment for at least 20 minutes and then attempt to reconnect.

We will monitor the connections and report how this is going in 1 hour (14:52)

Thank you for your patience.

----------------------------------------------------------------

We can see users are now connecting and believe the issue to be resolved.

BT are investigating the issue with the configuration file, once we have the explanation we will update this post (although the post will be closed).

Please accept our apologies for any inconveniences caused during this issue.

186k Support,