“Who uploaded this?!”
Posted on by Daan BroekhofWe have all made this mistake at one time or another in our web-developer careers: uploading the wrong files to a production webserver.
“Ey, it happens”, you could say. A lot of the time the aware software engineer picks up the obvious error right away, and fixes it before anyone notices.
Yet on many other occasions they sneak by unnoticed until they break down the application on the most unfortunate of times. Like that one make-or-break presentation.
There are many variants on this ‘wrong file’ theme, from partial file upload, to accidentally overwriting online-only configuration files and beyond.
It can and will happen to everyone: the-one-man developer or many-headed expert dev-team, we’re all in the same boat of trying to transfer our precious application state from our (preferably stable) development or staging location to the mission critial and mystical ‘production environment’.
This process of placing a version of your application online and preparing its state (database changes / etc) is part of what is called Software Deployment.
Managing this process, making sure that it is kept neat and tidy, with ways in, out and back, is often what distinguishes the smooth ‘no-one noticed’ upgrade from the emergency weekend-long upgrade-correction sludgefest.
So what can we do? A list, in order of importance.
- Revision control – have your code in a repository from where you can get old versions of your software. Know who changed what and when. Many articles exists about this, and if you’re not doing this, even as a one-man development team, start doing it right now. No exceptions.
- Versioning – when you put your application online, you want to know which snapshot of your application that is. Make sure to make such a snapshot (a SVN tag for example) every time you upload. This may look like a nuisance, but one year from now, you’re may not be able to find back that specific state, if you do not mark it. Combine this with a clear change-log per version and you can easily communicatie what just happened to your client/boss.
- Script your deployment – don’t go through all the deployment processes yourself. You are a weak human, who is slow and may make obvious errors. Instead, use a script with distinct steps, stopping at any unexpected results. Pick an existing task-system off the open-source shelf or roll your own little script. Make it specific for a location but not a version. Don’t try to generalize it too much. I could say alot more on this subject, but will keep it at this.
- Upload different versions of your code – You don’t have to have ‘only one’ version on your servers. You can have the last few versions present if needed, with the current ‘live’ version being the target of a symlink (or equivalent) from wherever your have configured your webserver to look for your application files. So after you are done uploading your version – you can switch to the appropriate version by just changing that one symlink.
This will also allow you to switch back to an old version quickly, and prevent access to a half uploaded code base. Using symlinks, you can even reroute your application to a static ‘under maintenance’ page while you do all your critical stuff. Done by your deployment script, of course. - Put your configuration files in your revision control system – this may seem a bit specific, but it is worth a lot to know who changed the configuration and when. You may want to keep these configuration files in a different location in your revision system. Name it after the location it is for. Make the deployment script always use it.
- Do not upload your local development files or directories – “What? That’s exactly where I have proven it to work!”, you may claim. But that one uncommited change will get you every time, if the configuration or test variables will not… Let your deployment script pull the files directly from your revision control system.
- No online editing – the one thing that can screw up a neat deployment setup is going ‘rogue’. That quick fix, or critical typo. It is worth a version or build number. Those are cheap. If you *really really* need to manually put something on your production server, for those seconds-or-death bugs, always deploy a neat version later with the same fix – as soon as possible.
- Log your deployments – Put a few commands in your deployment script, which log the start (and stop!) time of a deployment. It can be as simple as an email to deployment@yourcompanyname.com. Auto-relay that to your team-members. It keeps everyone up to date.
And that deployment which started but didn’t stop? Alarm bells should be ringing.
Those are the basics. There are many more aspects which I have not gone into, like:
- Database data & schema changes - How do you do database changes? Can you revert back data? Do you want to?
- Local deployments – Why not use your deployment scripts for setting up your development enviroment? Set that new team-member up in minutes instead of a day.
I do not claim to be an expert on this subject, nor do I claim infallibility, so please discuss!
