[Done]RAM upgrade on Web70 on Saturday

Posted in Scheduled downtime by

Web70 will be taken down on Saturday August 3rd at 6am UTC for a RAM upgrade. The expected downtime is around 20 minutes.

-
-

[Fixed] RAID problems on Web400

Posted in Downtime by

Web400 has been taken down for maintenance due to problems in RAID cabling which is being replaced.

We will update this post as the maintenance progresses.

[Jul 28 09:27 UTC] The server is back with new RAID cabling.

-
-

[Fixed] Web311 inaccessible

Posted in Downtime by

The server has lost all network connectivity after we had to reboot it since it became unresponsive.

We are following up with the datacenter now to get this fixed as soon as possible.

[08:12 UTC] On further investigation, we found that the disk being read-only was the cause of the server losing network connectivity so we have

rebooted it into rescue and started a fsck.

[08:22 UTC] The server is back after the fsck and OK now.

-
-

[Fixed]Network maintenance affecting several servers

Posted in Problems by

Network maintenance in one of our data centers is affecting connectivity and causing slow DNS lookups for many servers. We’re working to restore service and hope to have this issue resolved soon.

2013-07-24 17:21 UTC: The data center reports that the issue is resolved. We’re monitoring services to confirm and will update this post when we’re satisfied that service is restored.

2013-07-24 18:49 UTC: We’ve confirmed that connectivity has been restored to all servers.

2013-07-24 19:14 UTC: We’re re-opening this issue because there seems to be some lingering problems related to resolving DNS. We’ll update this post when the issue has been corrected.

2013-07-25 05:28 UTC: We are still working with the data center to fix routing errors that are happening sporadically.

2013-07-25 20:57 UTC: The datacenter has found work arounds for the issues that were causing sporadic routing errors. All servers should now be online and functioning normally.

-
-

[Done]Scheduled file system maintenance on Web391 between 03:00 and 07:00 UTC on Friday, 26 July 2013.

Posted in Downtime by

On Friday, 26 July 2013 between 03:00 and 07:00 UTC we will take this server offline to run a manual file system check. We will update this post as the maintenance progresses.

2013-07-26 03:00 UTC: We are now taking the server offline to perform the file system maintenance. We will update this post as the maintenance progresses.

2013-07-26 03:17 UTC: The maintenance is finished and the server is now back online and functioning normally.

-
-

[Fixed] Slow performance on web337

Posted in Downtime by

We are currently working on web337 to determine the cause of sporadic slow performance. The issue appears to be disk-related, although database activity is also affected.

We’re doing everything possible to stabilize the server at this time, and will update this post as soon as we have more information.

2013-07-20 20:54 UTC: There are file system warning still being investigated, but the performance issues have been resolved.

-
-

[Fixed]DDOS on Web162

Posted in Downtime by

Web162 server is experiencing a DDoS attack which our DDoS protection system is having trouble mitigating.

We’re doing everything we can to restore service at this time, and will update this post as soon as we have more information.

2013-07-18 01:03 UTC: The amount of traffic has decreased to normal levels. We’re still watching the bandwidth closely to insure that the attack is over.

-
-

[Fixed]Web313 down

Posted in Problems by

Web313 is down following a kernel upgrade. We’re working to get the server back online asap.

Update: We managed to fix the issue within 5 mintues. The server is back to normal.

-
-

[Fixed] Web213 inaccessible

Posted in Downtime by

We are investigating the issue and will update as soon as we have more information.

2013-07-12 17:03 UTC: We are currently running a file system check.

2013-07-12 18:19 UTC: The filesystem check is 58% complete.

2013-07-12 18:55 UTC: The server’s file system errors are currently unrecoverable. We are preparing to restore the machine from backups and any data we are able to recover from the current server. The server’s new IP address will be 108.168.213.84.

2013-07-12 19:51 UTC: The new server has now been provisioned and is currently in the process of installing our platform and tools.

2013-07-12 21:08 UTC:  The recoverable data from the old server is now being transferred to the new server. While the old server’s file system is damaged beyond being able to boot there is a large amount of good data on the machine and we are recovering as much as possible.

2013-07-12 23:17 UTC: The data recovery is ongoing.

2013-07-13 00:59 UTC: Approximately 30% of the data has been restored to the new server.

2013-07-13 03:09 UTC: Approximately 42% of the data has been restored to the new server.

2013-07-13 04:07 UTC: Approximately 52% of the data has been restored to the new server. We are currently in the process of restoring all PostgreSQL and MySQL databases.

2013-07-13 07:43 UTC: Approximately 90% of the data has been restored to the new server.  The databases have already been restored.

2013-07-13 12:18 UTC: We’ve restored all data from backups. If you notice any problems with your account just open a ticket and we’ll look into it ASAP.

 

-
-

Reminder: Outgoing SMTP server to discard mails using well-known domains in senders

Posted in Downtime by

Effective now, our outgoing SMTP servers will stop relaying outgoing messages which use well-known domains like gmail.com,yahoo.com, aol.com etc in the sender address. Messages sent using such domains in the sender address will be silently discarded.

This is being done as most of these domains use SPF and DKIM, hence relaying mails for them decreases our SMTP server’s reputation and increases chances of legitimate mails getting misidentified as spam.

If you have any questions or concerns regarding this change, please open a support ticket via our control panel to let us know.

-
-