Yesterday we announced unexpected server downtime and maintenance for a download server used by some of our applications, most notably our technician tools such as d7 and d7II.
These issues came earlier in the week when our hosted server suddenly went offline for more than an hour in the night; although it appeared to recover we had a new IP that prompted some DNS changes, but all was still not well. We then discovered that a configuration from last year had been automatically restored during the first downtime, along with older app data used by some Foolish IT applications. Of course there was a recent backup and early morning on Friday we rolled that out, however by Friday afternoon the server was not reachable again for an undetermined failure, and wasn’t replaced with another server until later in the evening.
After the most recent replacement from our backups, all appeared well with the exception of some configuration data lost in the restoration process affecting the d7II “SFX Mini” remote deployment tool and it’s ability to launch a remote copy of d7II, which continued late into Friday around midnight ET when the issue was resolved and tested.
Currently all issues are resolved since midnight (ET) on Friday, other than the initial mystery of these events which we’re still investigating.
Plans are in place to replace this immediately with a new server using no backups other than the data being served to our applications (in addition to the redundancy server we were going to add after code updates on our software can be done to accommodate additional backup selections.) Of course we’ll build and test the replacement server in parallel to normal operation to minimize or eliminate any potential downtime.
We deeply apologize for any inconvenience this may have caused!
The spawned idea: (What do you think? Yes or no?)
d7II “SFX Mini” self-hosting! Currently you can host your own configuration and definition files while using the “SFX Mini” for remote deployment of d7II, but it still relies on our servers for the most current binaries.
While complete self-hosting was planned as part of an optional management server for d7x collaboration across different copies and technicians, hosting your own binaries for the d7II SFX Mini could be a project we did first, devoting time to that right now, and this would take care of issues such as these our servers going down unexpectedly (provided your own servers stayed up.)
If we spent time on the project it would still be a small one, because adding a lot of extra features at this point would remove too much time from d7x development. We could easily include both FTP login using your existing self-hosted d7II cloud server, and also add direct download via HTTP/HTTPS with existing code, enabling you to use the new feature without FTP (though obviously you’ll still need a webserver.) We can even easily make it a ‘backup’ plan only, using our server if it is available otherwise. What we cannot do easily is AD/HTTPS/Webdav or other communication, login, or security protocols, or add other features I would most certainly prefer reserving to the newer d7x code base.
so…
Should we devote time now to an entirely self-hosted d7II SFX Mini deployment right now?
Tell us what you think! Visit www.foolishtechshow.com on Monday or anytime next week and drop your opinion in the IRC while we discuss it with you live on the show!
