Technical teams have confirmed there has been no errors since the rollback yesterday. All systems are now functioning as per BAU. Ticket has been raised to investigate the root cause.
No further updates
Posted Nov 21, 2024 - 14:30 NZDT
Monitoring
Update as at 20/11 18:00:
Remedial work has been completed . Technical team will investigate the logs tomorrow during the peak traffic time .
Next update will be around 21/11 12pm or earlier if available.
Posted Nov 20, 2024 - 18:06 NZDT
Update
Update as at 20/11 14:34:
Technical team will be carrying out remedial work from 17:00 to 17:30 tonight
Next update as soon as available.
Posted Nov 20, 2024 - 14:37 NZDT
Update
Update as at 19/11 18:04:
Technical team will continue to investigate the logs tomorrow during the peak traffic time .
Next update will be around 20/11 12pm or earlier if available.
Posted Nov 19, 2024 - 18:15 NZDT
Update
We are currently experiencing an issue with the Assurance website & Assurance API, since 15/11 09:56
Retail Service Provider (RSP – Customer) Impacts:
Testing of fibre and Copper ASIDs may intermittently return an error message stating "we are currently experiencing an outage and this data is not currently available", asking end users to try again.
Please Note: this is also impacting the ONT to RGW connectivity
Service Company Impacts:
Testing of fibre and copper ASIDs may return an error message .
Please Note: this is also impacting the ONT to RGW connectivity test.
Update as at 19/11 17:24:
Technical teams deployed a potential fix into the production environment. The errors have reduced but continue to be present. Technical teams continue to investigate.
Next update as soon as available.
Posted Nov 19, 2024 - 17:39 NZDT
Update
Update as at 19/11 12:36:
Technical teams continue to investigate further.
Next update as soon as available.
Posted Nov 19, 2024 - 12:44 NZDT
Identified
Update as at 18/11 17:28:
The issue has reoccurred. This has been escalated to technical teams to investigate further.
Next update as soon as available.
Posted Nov 18, 2024 - 17:37 NZDT
Update
Users may have noticed a recurrence of this issue between 12:20-12:48 today. However it self-resolved and subsequent testing has been carried out successfully.
This incident will remain under monitoring until COB today at this stage.
Posted Nov 18, 2024 - 16:41 NZDT
Update
The number of errors observed has now dropped off to a very low level.
This incident will remain under monitoring over the weekend, and technical teams will follow up and review on Monday morning.
Please Note: this is only affecting the ONT to RGW connectivity test.
Posted Nov 15, 2024 - 19:31 NZDT
Monitoring
Functionality to all impacted applications/systems has been restored.
Monitoring will continue until COB today to confirm stability.
Posted Nov 15, 2024 - 14:54 NZDT
Update
Technical teams have restarted a backend server but systems continue to be impacted.
Investigations are continuing.
Next update as soon as available.
Posted Nov 15, 2024 - 14:14 NZDT
Investigating
We are currently experiencing issues with the following systems/applications, since 09:56 this morning:
Retail Service Provider (RSP – Customer) Impacts:
Assurance website & Assurance API: Testing of fibre ASIDs may return an error message stating "we are currently experiencing an outage and this data is not currently available", asking end users to try again.
Service Company Impacts:
TechMate: Testing of fibre ASIDs may return an error message stating "error 429 - too many requests".
Technical teams are urgently investigating.
Posted Nov 15, 2024 - 12:23 NZDT
This incident affected: APIs (Assurance API), Service Partner Websites and APIs (Tech Mate (Testing Fibre Services)), and Websites (Assurance (Testing Fibre Services)).