Note. ns1, ns2, ns4 and ns5 are not affected.

13/8/2024 1530 The maintence window has been extended due to the electrical engineers having installation difficulties. Please note. ns1, ns2, ns4 and ns5 are not affected.

13/8/2024 1845 The maintence window has been extended due to on-site engineers having installation difficulties. Please note. ns1, ns2, ns4 and ns5 are not affected.

13/8/2024 2200 The maintence window has now been extended to the 14th. Any updates will be posted here.

Note. ns1, ns2, ns4 and ns5 are not affected.

8/8/2024 06:01 Engineers have identified an electrical issue affecting multiple racks in the datacenter.

8/8/2024 09:00 No updates at the moment as we are still waiting on the power companies engineer to arrive.

8/8/2024 12:45 Engineers are still working on the power issues to multiple racks in the datacenter. Please note NS1, NS2, NS4 and NS5 are still servicing DNS queries.

8/8/2024 18:30 Engineers have resolved the electrical issues affecting multiple racks in the datacenter. Our engineers are just running tests on the NS5 cluster before making it production ready.

8/8/2024 18:52 NS5 is is production more and all NS1 to NS5 services are handling DNS queries.

We are investigating an outage in one of our datacentres. This currently affects the outmail cluster mx496502.

24/06/2024 18:28 Engineers have identied a configuration issue with backend NAS. mx496502 has been failed over to another cluster and queues will return to normal.

24/06/2024 18:36 Engineers have resolved a configuration issue and mx496502 cluster is fully operational.

24/06/2024 19:10 Engineers will continue to monitor the cluster.

We are investigating an outage in one of our datacentres. This currently affects the outmail cluster mx496502.

22/06/2024 06:15 Engineers have resolved a storage hardware issue within datacentre and services and queues are returning to normal.

Closed

Closed

Note. ns1, ns3, ns4 and ns5 will not be affected.

Closed

During this window there will be some disruption to 3CX VoIP hosted services.

Closed

Note. ns2, ns3, ns4 and ns5 will not be affected.

Closed

Note. ns1, ns2, ns3 and ns4 are not affected.

22/2/2023 15:40 Engineers identified a configuration error resulting from a update process.  This has been resolved and this incident closed

Closed

There is a scheduled maintenance window for engineers in the Amsterdam data center starting from Thursday 24th November 2022 at 00:00 (GMT).

During this maintenance window ns2.dns-engine.com will be off line. Although there is a 6 hour window allocated it is hoped that services will be operational earlier. As always please keep an eye on the twitter feed and this status page.

Note. ns1, ns3, ns4 and ns5 will not be affected by this maintenance window.

24/11/2022 01:35 Maintenance window complete.

Some customers still on the legacy Profilter service will not be receiving email. There are issues in the datacentre that our on-site engineers are working on at the moment.

2022-06-17 0930 We are currently investigating issues with the legacy profilter platform and degraded performance. Emails are still flowing but not how we would expect them.

2022-06-17 0945 The legacy profilter platform was due to be retired at the end of 2022, as sush many clients have already been migrated to the current profilter service. A decision has been made this morning to migrate the remaining number of clients from the legacy profilter platform today. All clients affected will be notified this morning. For some domains there is no action required from the client and the migration will be automatic. For domains that do need changes made the client end our support team will contact you directly to discuss options.

2022-06-17 1150 The issues regarding the degraded performance of the profilter legacy platform have been resolved.

2022-06-17 1420 All domains that could be migrated seemlessly have now been completed.

2022-06-24 1730 All domains that were on the legacy profilter platform have been migrated.

Resolved

There is a scheduled maintenance window for engineers in the UK data center starting from Tuesday 31st May 2022 at 16:30 (GMT).

During this maintenance window there will be a slow performance due to high IO and for a short period an outage whilst the clusters are failed over to a new environment.

Note, If your service is hosted on either hosted-lon-3 or hoested-lon-4 then you will see issues with websites hosted and emails hosted on those clusters.

During the maintenance window regular updates will be published on our Twitter feed.

Postponed (see details)

There is a scheduled maintenance window for engineers in the UK data center starting from Thursday 26th May 2022 at 16:30 (GMT).

During this maintenance window there will be a slow performance due to high IO and for a short period an outage whilst the clusters are failed over to a new environment.

Note, If your service is hosted on either hosted-lon-3 or hoested-lon-4 then you see issues with web hosted and emails hosted on those clusters.

Update due to unforeseen circumstances at the data center, engineers have had to postpone the maintaince window till next week.  Tuesday has been identified as the next maintenance window with a backup window of Wednesday all the same time slots (1630 to 1800)

Resolved

On Monday 9th May 2022 at 23:00 (GMT) there is a scheduled maintenance window for engineers in the Amsterdam data center to install a new Automatic Transfer Switch (ATS) in the power delivery system.

During this maintenance window it will only affect the NS2 cluster, whilst each server in the cluster is shutdown and moved to the new power circuits.

Note, DNS services will run normally during this maintenance window as NS1, NS3, NS4 and NS5 clusters will not be affected.

Resolved

2022-04-01 0913 We are currently investigating issues with profilter inbound routing.

2022-04-01 0950 We are working with engineers in the London datacenter to resolve a routing issue.

2022-04-01 1015 We are continuing to work with on-site engineers who have identified an error in a cloud-init configuration which has rolled out across the datacenter.  This is currently preventing profilter clusters from failing over, so we are working on forcing a manual failover to another datacenter.

2022-04-01 1040 Incident resolved.  profilter is processing emails, all queued emails on the failover cluster have been processed as well.

Resolved

2021-10-08 1905 We are currently investigating performance issues with the Legacy profilter cluster. This incident only affects the Legacy profilter cluster and not the mx??25??.smtp-engine.com clusters.

2021-10-08 2330 We are continuing to work on the incident. A disk in the storage area has been replaced and the storage raid is currently being rebuilt. During the rebuild process we will still performance issues.

2021-10-09 0345 Storage area performance is still degraded due to rebuilding.

2021-10-09 0730 Storage area rebuild complete. Engineers will run tests on the cluster.

2021-10-09 0810 Cluster is running normally, will continue to monitor.

2021-10-09 1330 Incident closed.

Resolved

Resolved

03:05 Network connectivity lost to one of our datacentres - investigation underway

03:27 We have reports of a fire in the France datacentre.

04:15 We are provisioning failover of affected clusters to a separate datacenter and will re-route traffic through there.
    - Affected cluster nodes are NS5 and profilter (MX332501)

05:30 The datacentre in France is split into multiple units.  One unit has been destroyed due to a fire and the other three buildings are currently unavailable due to fire crews complelety isolating the industrial estate of power.  Usually a cluster node would failover to the sister location but this isn't possible at the moment so engineers are working on a manual failover to the UK datacentre.

06:07 Status on MasterDNS and BackupDNS changed to yellow as global coverage of DNS records isn't affected.  ns5 is currently off-line but ns1 to ns4 are fully operational.  Failover of ns5 to another datacenter will have an impact on IP Address, so engineers are waiting to see if units 1, 3 or 4 in france will be released by the fire crew first in order to preserve IP Addresses. Please continue to watch this status page for updates.

06:45 The fire at the datacenter has destroyed building 2, building 1 has some fire damage.  Units 1, 3 and 4 will remain offline until the fire crews release the scene, which will not be until later on today.  Therefore ns5 will remain offline.  proFilter (MX332501) failover engineers are working on testing the new cluster before switching on live.

08:30 profilter cluster (MX332501) has now been successfully failed over to (MX442503), and emails flowing again.

09:08 profilter status changed to yellow.  No updates as yet to extent of the fire damage. Please keep an eye on this page for status updates

11:30 Incident Update, at 00:47 this morning a fire started in the France datacenter where we (Prolateral) have several clusters of services sited. We weren't aware of the incident until 02:58 this morning when our own network monitoring started to alert us.  The alerts were due to the fire service cutting all power to the site at 02:54.  Sadly by 04:09 this morning the fire had completely destroyed one (Unit2) of the three units.  We are grateful that there has been no loss of life to either datacenter staff or the fire service in attendance.  By 4am this morning we invoked our disaster recovery plan because it was clear that automatic failover of the clusters wasn't going to happen, so engineers worked on adding additional clusters to the UK datacenter and then forced a manual failover.  Between 06:00 and 08:30 clusters were migrated over and back online.

11:35 Incident Update, As previously mentioned unit2 was lost, there looks to be partial fire damage to unit1,  Unit3 and unit4 are ok, just no power at the moment.  As soon as the fire service start releasing parts of the site to the datacenter staff, engineers can further assess and start to power up those buildings.  For your information we have several clusters at that location in various units.

20:00 Incident Update, Datacenter engineers are planning on restoring power to Units1, 3 & 4 by Monday 15th.  Our thanks to all those engineers working day/night to get things back to operational as quick as possible.

23:00 This ticket will remain open throughout the incident (hence the yellow status flags above) of the datacenter, however all affected (Prolateral) services were failed over to the UK Datacenter this morning (see above) and fully operational.

2021-03-15 Update.  It has been confirmed that all clusters hosted in the datacenter have sadly been losted.  Over the next two weeks we look to replace those clusters and add back to the swarm.  This incident will remain open during this time.

2021-04-01 Update on NS5. We are working with the datacentre team in restoring the NS5 cluster. However whilst electrical work is continued in the Units it's not reliable enough to make a production service, as such NS5 we remain failed over to the NS4 cluster. More information to follow on this.

We apologies for any inconvienance caused and we are working hard to resolve the situation as quick as possible.

Progress will be reported here and on our Twitter page.

2021-04-07 Update on NS5. The cluster has been restored and it currently under going testing before going into production ready status.

2021-04-08 Update on NS5. The cluster is live.

2021-04-10 This incident will be closed.

11:50 Network connectivity lost to datacentre - investigation underway

13:30 Network connectivity in datacentre is being worked on by provider 

14:30 We are provisioning another cluster in a separate datacenter and will re-route traffic through there.

16:20 Engineers are working in three teams.
    - Team 1 - working with the datacenter engineers that is suffering a complete outage
    - Team 2 - working with the engineers of the sister datacenter to force a failover scenrio
    - Team 3 - working to bring up a new cluster to enable emails to flow from the failed datacenter

17:15 Engineers are still working to resolve the issue.

18:20 Engineers are starting to bring up a new cluster.  Once internal testing has been completed the cluster will go live and be ready to accept email again. Any emails that you are trying to send will be queued on your server ready and waiting to be sent out.  This will happen naturally once the cluster is made live.

We apologies for any inconvienance caused and we are working hard to resolve the situation as quick as possible.

Progress will be reported here and on our Twitter page.

18:55 The issue with the network equipment and configuration at the Datacentre has been resolved and the mx496502 cluster is fully accessible. Queues are clearing. 

08:30 We are experiencing connection issues to MX496502.  Engineers are working on the problem.

08:35 Engineers have identified the cause and are working on a resolution.

09:15 Engineers are still working on the problem.  Due to the earlier (unrelated DNS issue) the cluster failover hasn't worked.  Engineers are working with the datacenter engineers to resolve as quickly as possible

10:21 The root cause has been identified and engineers are working to resolve the problem as quickly as possible.  

11:18 Service to the cluster MX496502 has been resolved.  Email to that cluster is now flowing. 

06:05 We are experiencing MasterDNS and BackupDNS issues.  Engineers are working on the problem.

06:40 Engineers have identified the cause and are working on a resolution.

07:50 All DNS services have been restored.  The fault was due to a cascading configuration error which has been resolved.  Engineers will continue to monitor

11:40 One of our datacentres is currently affected by an ongoing network-outage. More information will be reported here.

12:31 Network routing issue within datacentre now resolved.

A major datacentre outage in our Strasbourg datacentre is currently being investigated.

11:30 update. A total power & backup power outage has affected 3 datacentres and thousands of servers. Work to restore continues.

13:10 update. Power was restored, but we are awaiting datacentre reboot of networks as the systems are brought back online. 

19:20 All IP routing issues resolved and full connectivity to all Profilter servers restored.

Scheduled

Resolved

Microsoft (Outlook, Hotmail, Live.com, etc.) blocked a significant range of IP addresses. We are rerouting email via unaffected IPs while the issue is resolved.

Update: re-route fully in place by 11.15. MS fix ETA unknown.

Update: blocks removed and normal routing resumed.

Resolved

Scheduled

Resolved

Resolved

Network routing and DNS issue prevented hostname lookups. DNS server issue resolved. Queues now clear.

Resolved

Network outage at data center has caused queues to build. They are clearing and should be clear within the hour.

Resolved

Significant Spam attack inbound causing delays on email. Servers rebalanced and queues are clearing.

Resolved

Some Profilter customers are expirencing connection problems to a selection of profilter clusters. This is due to the datacentre provider having issues within that centre. System engineers are currently on site and working on the problem.

Update 17.20: Datacentre outage is more severe than first estimates for resolution.  Outage is affecting a large number of european subscribers.  We (Prolateral) are working with datacentre engineers to find a speedy resolution.  More updates to follow.

Update 20.05: Datacentre network component failure has been resolved, routing rebuilt and is back online. Queues have built up and are clearing steadily, otherwise all services running normally.

Resolved

Some customers are seeing bounces due to black-listing, we are migrating those services away from affected IP addresses.

Resolved

Scheduled Maintenance work in the LondonCloud. 

Please be advised there is scheduled maintenance on several servers in the LondonCloud (hosted-lon-1, hosted-lon-2 and hosted-lon3).  The maintenance window is very small to allow for the servers to be rebooted following the maintenance work.  This work will affect web hosting services only for approx 15minutes. The small outage will occur between 2300 and 0400 Thursday 12th Sept 2013.

Resolved

Some customers are still experiencing problems sending emails to AOL, Demon Internet and EmailSrvr domains.
It appears these Internet Providers are using a cached database from CloudMark CSI and not a realtime lookup.
Therefore this is a knock on effect from yesterdays problem which was effecting a small number of outMail Clusters
Engineers are working on resolving the issue.

Update: Engineers have resolved the issue

10-09-2013
07:35

10-09-2013 07:45

Resolved

Some outMail clusters are experiencing email delivery problems. Engineers are working on the issue now.

Update: It appears CloudMark CSI have wrongly blacklisted some outMail Clusters, this has been resolved with CloudMark and all outMail clusters are blacklist free

09-09-2013
10:00

09-09-2013 15:20

Resolved

Some outMail clusters are experiencing email delivery delays. Engineers are working on the issue now. 
Engineers have resolved the issue and queues are running normally

05-04-2013
11:50

05-04-2013 12:21

Resolved

Some reports of time-outs when accessing pages. We are investigating possible network routing issues at one of our providers.

Resolved

Database maintenance affecting access to some webservers and outmail intermittently.

Resolved

Some customers are reporting delays through the outmail cluster. Engineers are investigating.

DDoS SPAM forced re-routing of email queues. All queues now back to normal.

Resolved

Some outMail clusters are experiencing high volumes of connection requests which is causing some email delays of upto 5-10minutes.

Actions have already been put in place and we are monitoring the situation.

Scheduled

Resolved

Scheduled Maintenance work in the LondonCloud. 

Our Tier1 provider is conducting maintenance to their services.  We expect no disruption to our services but this is purely an advisory to our web hosted customers.

Scheduled

Resolved

Scheduled Maintenance work in the LondonCloud. 

Please note there will be a short period of downtime which will affect some web hosting, email hosting customers.  This will also affect access to the CI Portal during this time.  This is due to maintenance work being carried out on a core edge firewall.