June 21, 2014
We are currently going through some server and network related maintenance for our CMS Websites (Faithwebsites), Auction Trak and Donor Trak applications. This maintenance has been causing intermittent and very short term outages, which we anticipate may occur over the next 48 hours. We apologize for this inconvenience and are working diligently to avoid any outages if at all possible.
If you have questions or concerns, please email support@diswebsites.com or start a new ticket from your support center.
UPDATE, June 23:
As the comments below indicate, we were stable on Monday and then had our scheduled maintenance starting at 8:00 pm CT. All services were fully restored already by 8:40 pm Central Time. Please see below for further details.
UPDATE, June 25:
We are still experiencing intermittent problems for our Site Manager (websites), Auction Trak and Donor Trak customers. We now know that this is in large part due to a Denial of Service (DoS) attack on our server. Our security has not been breached, but in protecting your websites and data, our server is simply "tiring" and shutting down at intervals. We are battling the attack and shutting down access as soon as we can safely determine the malicious traffic, but it is not an easy feat.
What we have done is secure all your sites and content, and we have measures in place to ensure that the outage - if it does happen - is very short in duration. So short, we hope that it goes without notice. Of course, at times, some of you are catching the system down.
We will continue to update this thread as more information is available. Know that we are doing everything we can to fully and quickly resolve this situation.
Monday, June 23, 8:15 am CT
UPDATE
At this time, the network / server maintenance that we had hoped to be completed over the weekend was not successfully finished. As a result, we are still experiencing short term and intermittent outages.
These outages are short in duration and may not even be noticeable at times.
We are sorry about this interruption of service and thank you for your understanding. Know that we are continuing to make this our top priority and hope to have all the problems and needed maintenance done and resolved very soon.
Monday, June 23, 4:45 pm CT
UPDATE
We are now scheduled for a maintenance outage and believe that with this upgrade, this will be the last of the intermittent outages. This planned maintenance window will take place tonight between 8:00 pm Central Time and 10:00 pm Central Time. During this maintenance window, we anticipate a interruption of service both with Site Manager access, Auction Trak and Donor Trak admin access AND with your website.
Monday, June 23, 10:15 pm CT
UPDATE
Our upgrade and server maintenance was completed in short order; all services were fully restored by 8:40 pm Central Time. We are not experiencing any further issues and anticipate that the maintenance and upgrades are now in place and no further interruptions are expected.
Thanks for your patience and understanding during these required server and network upgrades!
UPDATE, June 25:
We are still experiencing intermittent problems for our Site Manager (websites), Auction Trak and Donor Trak customers. We now know that this is in large part due to a Denial of Service (DoS) attack on our server. Our security has not been breached, but in protecting your websites and data, our server is simply "tiring" and shutting down at intervals. We are battling the attack and shutting down access as soon as we can safely determine the malicious traffic, but it is not an easy feat.
What we have done is secure all your sites and content, and we have measures in place to ensure that the outage - if it does happen - is very short in duration. So short, we hope that it goes without notice. Of course, at times, some of you are catching the system down.
We will continue to update this thread as more information is available. Know that we are doing everything we can to fully and quickly resolve this situation.
Update: 4:00 pm Central Time
We now have an additional engineering and network team working on this situation. We expect that there may be slow-downs in the system through the evening, but we should be avoiding any actual outages on your live websites.
It appears that the DoS attack is "intelligent", a term which is used when the bots adjust to your preventative measures and renew their attack on different parts of your server.
Again, your data is backed up nightly and moved to a completely separate server - physically different than the primary server. Also the DoS attack has not affected any content, data, emails, etc. It has flooded our servers with traffic that has overwhelmed the system, resulting in the slowness and/or outages you have experienced.
Please email or chat support if you have any questions. If you are struggling to get a larger file uploaded, contact support@diswebsites.com. We might be able to upload it to your Site Manager for you and then you can link and update your site from that point.
UPDATE, June 26, 6:30 am
Our troubles continue. With another upgrade and yet another engineering team brought into consult, the solutions resulted in an outage this morning. We know this is not the service you require of us, and for that we apologize. We continue to do everything in our power to restore stable, 24x7 uptime.
Update, June 26, 7:30 am Central Time
We continue in a state of outage right now, although some services have been restored. We are in constant communication with our cloud hosting provider and all technical and engineering teams and are praying we are able to bring the systems and your websites back up soon.
Update, June 26, 9:40 am CT
As of this moment, you may have service restored but we anticipate that the service may still go up and down some.
We have identified that it is a large scale attack on our network (a Denial of Service or similar). Our networks firewall and other protections have been keeping the attack at bay, and the attack is sophisticated enough to have hidden itself within the network, resulting in the inability to quickly isolate and purge.
While the teams work to track and remove the source of the DoS, they may need to take us down - or worse - the attackers may be successful in taking us down while our teams try to work on other areas of our network.
The good news is that your data is safe. Our network securities did what they were supposed to do - and in a very real way sacrifices the equipment and took us down in a data-safe way before allowing unauthorized access to any of your sites, or your curriculum, donor or auction data.
The bad news is that this is not going to be a fast fix to purge. We do not have an ETA, but we've been assured that the crews are doing everything they can to keep us up while they work.
You are welcome to access your application and/or website -- we do anticipate 80% to 95% up time today, despite the attacks and the work to remove those attackers.
If you have questions or concerns, please email support@diswebsites.com. Thank you for your continued patience.
Update, June 26, 10:05 pm CT
Customers were sent an email several hours earlier today with a full explanation. You can read more about this attack and how it is affecting us - and ultimately your service here.
At this time, all services are remain fully functional and stable. It appears that the blocks and added security measures have been successful. We are not sure that we have fully neutralized the attack, but do feel that we have reached a more stable point.
Update June 30, 5:30 am CT
Well, we had expectations we were through the woods with this but we did go down in the middle of last night, and at this moment are still down right now. We may not have neutralized the attacks as we had originally thought on Friday afternoon.
Please stand by, again, teams are working on this and we expect to be back up soon.
Update, June 30, 6:25 am CT
And our servers are back up now, so your websites and admin access should be working now. We do not have updates from the technical teams at this time.