Spree 1.3 to Spree 2.1 - An Upgrade Odyssey
Note: a few days after I published this post, the spree team launched a new website. For the moment you have to visit a specific url to get to the spree deployment service: http://spreecommerce.com/stores. Clicking on a store name takes you to a page where you can select the deployment service under the Add Ons heading, and from there follow my instructions in this post.
I’ve also noticed that the team has published an ansible configuration guide. You may prefer to use that if you like ansible, or like having more direct control. There’s a manual configuration guide as well.
Before I get started, if you want to follow along with this post, you’ll need to install the following:
- ruby (preferably 2.0)
If you haven’t worked with git before, you’ll need to do some basic configuration. It will give you directions the first time you try to use it, so I won’t go over it here.
Also, do your homework on extensions before even thinking about the
upgrade. Visit the project pages for all of your extensions and find out
if they support 2.1. Many extensions don’t keep up with spree or play
catch-up months later. There will usually be a branch of the extension
in git which tracks the spree version, such as
spree-2-1. If you’re uncertain, contact the developer and ask. If
there’s no 2.1 extension, ask yourself whether you can live without it
or be prepared to wait for the developer.
So this will be a long one. Don’t worry though, if you work seriously with spree, it may save you boatloads of time as it has for me.
I’ve been working with Spree Commerce since version 0.60, about three years by now if I’m estimating correctly. If there’s anything that’s hard about Spree (and Rails in general), it’s upgrades.
So how do you go about tackling such a task? Let me tell you about my situation.
I run a simple, low-volume store. In fact, you can take a look at it at http://store.diditbetter.com. You can see the code for my store at https://github.com/binaryphile/spree-dibs-2.1. There’s a good readme which goes into some detail on what’s in there, but I’ll go over the basics here as well.
My store tries to stay as vanilla as possible, precisely to make it easier to upgrade. I’ve learned that many plugins get upgraded on their own timetable, if at all. If you want to stay up with the state of the art, you’ll stay away from frivolous plugins and only use what you need.
I use the basic spree theme, which is quite attractive. There’s a spree-fancy theme as well which I haven’t tried but is popular too. I make a bunch of small modifications to the theme, things like changing the logo, removing the spree tag in the footer and adding our Authorize.net seal to the checkout. All of these are basic modifications perfectly suited for using Deface overrides.
I also customize the emails with our branding and add html versions so Outlook users (our primary customers) get the email experience they expect.
Since we only sell non-physical items (software and services), we don’t have need of the delivery step in the checkout. In the past I’ve removed that step from the checkout process, but have had to reinstate it because we use the shipping address for other purposes. I’ve simply configured a zero-cost shipping option instead.
I have a handful of plugins:
- Print invoice
- Email to friend
- A custom referral-tracking module called “dibs-referral”
The print invoice plugin allows us to generate nice-looking pdfs from our orders.
The email to friend plugin adds a simple form which can be used to send an email from any of the product pages.
Since we don’t sell physical products but have lots of variants, the flexi-variants plugin allows us to manage all of the options from the master variant without having to create an explicit variant for every option combination. This is a lifesaver and we wouldn’t be able to run our store without it.
Finally, the referral extension allows our agents to send store links which prepopulate a cart for the end customer, including a session variable which ensures that the agent gets credit for the sale. The customer can also explicitly choose the referral agent in the checkout step.
There are lots of additional tweaks and fixes in the app and in fact, some of the referral functionality is overridden or extended in the app because there were parts which didn’t seem general-purpose enough for us to put in the plugin, which is also open-source.
We run our own production environment in-house, so we don’t use any kind of cloud service. Spree is pretty light on resources, so I usually feel it’s over-specced at a couple gigs of RAM and four processors. YMMV.
I use the Spree Deployment Service to configure the production box. The deployment service is awesome and very nearly turn-key. I’ve found a few items I’ve had to tweak for my environment, but I can’t recommend this tool enough. It doesn’t seem to get its due credit in the spree community.
Production runs on Ubuntu 12.04 LTS, which is the only platform supported by the deployment service. Ruby is version 1.9.3. I’ve tried using the deployment service to configure ruby 2.0, but the puppet scripts fail with 2.0. Rather than chase the issue I just decided to move forward with 1.9.3 explicitly.
The web server is unicorn, fronted by nginx. I use the deployment service interface to handle the details of configuration.
I configure all of these on a single app server. Like I said, I have a small store. The exception is that I use a different proxy server to front the store which I configure separately, so I don’t really use the nginx instance given to me by deployment service.
For deployment, the Spree Deployment Service does you the enormous favor of configuring Capistrano to handle it for you. As a newcomer to Capistrano, you won’t find adequate description of how it works in the deployment service docs, but you’ll get plenty of usage examples here. Capistrano makes deployment exquisitely more manageable and you shouldn’t miss it.
In development, I clone and run my store code directly on my system. I won’t go into the setup of the requirements, which is amply covered in the spree Getting Started guide. You should be able to get an idea of most of it from this post anyway though.
The Development Cycle
My normal development cycle is to work on code using sample data on my development machine. I can even develop adequately on my Chromebook this way, using the crouton sideloader for an Ubuntu environment. Once I’ve gotten everything working with spree’s sample data, I switch the database over to a copy of my production data and test that everything works there as well. This allows me to separate data and migration-related issues into a separate step, one where I know that the data is at issue and not the entire upgrade.
While I work, I’m using a git branch called “staging”. This allows me to drop what I’m doing at any time, in case there’s a bugfix or minor improvement which needs to be done to the store without having to wait for my development to finish. Once I’m done testing in development, I merge staging to master and test again before deploying to the production server.
Actually, there’s one more refinement to that process, one which I recommend. An unfortunate truth is that I’ve frequently run into issues during the deployment phase which didn’t show up in development. Sometimes it’s due to the deployment process itself and sometimes its because problem behavior doesn’t become evident until working with real, non-sample data, or even the due to the configuration of the production machine itself. Sometimes real data doesn’t respond in the right way to migrations, sometimes plugins have altered what the developers would normally have expected of the data. You don’t want these issues to only crop up once you’ve told everyone that there will be an upgraded store tomorrow.
For this reason, it’s tremendously useful to set up a staging server, one which goes through the same deployment process as production, runs the same database system and, importantly, runs the migrations on actual store data. I’ll go through the steps to set up a staging server relatively painlessly. I use Vagrant on my development box for this purpose. That obviously can’t be done on a Chromebook, but it’s nice to have this step take care of those concerns so that my normal development can be done in a lightweight environment.
The Upgrade Process
That brings me to the upgrade process itself for which I take, well, an alternate route. It’s more circuitous, which is why I’m documenting it like this, but what you get in the end is a known quantity.
Why do I imply that the regular process gives you an unknown? Because what you get isn’t the same as what someone else gets and especially isn’t the same as what a new spree store gets. If you follow the directions (which now seem to be regularly provided with spree upgrades in guide format, hooray!), you’ll very likely get a working store after maybe dealing with a few hiccups. Definitely in fewer steps than I’ll outline here.
What you won’t get is the same thing that a brand new spree user does. Your project will be different. It will be generated from the Rails templates supplied when you initially created the store in the way-back. Those files don’t stand still with each new Rails release, sometimes changing significantly. But what the spree guides don’t (and shouldn’t) do is take you through every possible upgrade path for the underlying Rails files. There would just be too many possiblilities. Instead, they tell you the changes which have to be made or else the store won’t work.
Call me a control freak, but there’s a reason for my madness. By going through the effort of bringing the store completely up-to-date, more up-to-date than the guide process, I’m moving myself from one class of user to another. I’m staying with the majority of future users. I can count on the fact that in the future, the spree community will be growing and that there will be more users who’ve created a store on the newer version than the old version with which I started. By keeping my store in line with what the new users have, I’m keeping my store more supportable by being in the mainstream. I’m less likely to experience oddball errors due to crufty configurations long abandoned by the Rails team. I’m willing to go through the effort to keep myself positioned for future support and to avoid the compounding of cruft upon cruft that comes from a normal upgrade.
The regular upgrade process typically involves updating your Gemfile to the latest versions in the appropriate places, then bundling and finally installing and running migrations. This, along with a few edits for key files in your app, represents most of the process. While I’ve never had one of these work as advertised (I always get migration issues), I wouldn’t blame you for skipping my process and going that route. More power to you.
My process involves creating a new spree store, migrating the data and then exporting/importing it into the new store. That way everything is spanking new. The rub is that the data may need some massaging, especially if you’re ditching plugins along the way, but what you end up with is almost exactly a new store. And that’s where I like to be.
The App and the Repository
Let’s start setting up the process. I’ll do it in a way where you can follow along with your own sample store and go through the whole process, minus whatever of mine you want to skip.
First thing we’ll need is an “old” store. I’ll set one up without any plugins, just spree 1.3. I assume you have ruby 1.9 or 2.0 installed. This is on my regular development machine running Ubuntu 13.10:
Modify the sqlite3 gem and add the mysql2 gem (or see 1.3 Gemfile gist):
Find the unicorn line and change to:
Find the capistrano line and change to:
Also add these lines at the bottom:
Exit and run:
Once that finishes, you’ll have a working spree development environment running on sqlite. SQLite is good enough for our purposes here. For MySQL, you’ll have to follow the directions in the spree guides.
You can test the store by running
bundle exec rails s and then
pointing your browser at http://localhost:3000/ once it has started
Ctrl-C stops the web server.
Wasn’t that easy? Amazing!
Important Bundler Commands
bundle install- installs gem versions locked by Gemfile.lock, or just the latest acceptable versions of everything if there is no .lock file. When in doubt, this is almost always the safe choice.
bundle update [gem]- update a single gem and its dependencies to the latest acceptable version, not updating other gems unless required
bundle update- update all gems to the latest acceptable version. Be careful with this command, it tends to break things. You can always revert Gemfile.lock and
bundle installto undo this.
bundle exec [command]- run the command using the Gemfile’s gems as the environment
Setting up Git
Now let’s set up a repo:
Add the following lines to your
.gitignore (or see .gitignore gist):
Save and exit. Next run:
You’ll also want to set up a
store-1-3 repo on github. Go do that then
Use your own username instead of “username” of course.
The Deployment Service
Now let’s set up a production server for the 1.3 store.
In order to do this, we’re going to use vagrant rather than a true VPS in the cloud or anything like that. So this will be a “fake” production server, just so we can go through it together.
Make sure you’ve got vagrant and VirtualBox installed and working. You should also install the guest additions plugin using:
store-1-3 directory, run:
Important Vagrant Commands
vagrant status- status of the current vagrant machine(s)
vagrant init- create a Vagrantfile
vagrant up- start a vagrant machine, or all machines in the Vagrantfile
vagrant up production- start the vagrant machine called “production”
vagrant halt- stop a vagrant machine
vagrant reload- reboot a vagrant machine
vagrant destroy- delete a vagrant machine from disk
vagrant box list- list local boxes
vagrant package- create a package.box file from the current machine
vagrant box add boxname [url|local file]- add boxname to your boxes
vagrant box remove boxname- remove boxname from your boxes
Vagrant runs everything headless, so there’s no need for a virtualbox
gui. However, you may want more visibility than
vagrant status affords
sometimes. What if you don’t know which machines you left running?
You can always open the virtualbox gui while vagrant machines are running. You won’t get a console session with the running machines, but you can see their status and perform operations such as powering down.
The only time you need to make sure virtualbox gui is closed is when you’re mucking with machine ids as I note in the aside on dropbox.
While that’s going, you can start configuring the deployment service. Go to http://spreecommerce.com/ and login (register if you haven’t already).
As I note up top, you’ll need to visit http://spreecommerce.com/stores directly to manage your stores for the moment. Looks like this part of the new spree site is still under construction.
Once you’re logged in, go to the My Account section and click the Stores heading. Click New Store.
Create a new store and call it “store.mydomain.com”, using your domain instead of that one. The url is arbitrary, just use “http://store.mydomain.com”. Label can be “production”. Click the Register button.
Now under Add Ons, click Add Deployment Service.
Leave the environment as “Production”. For Ruby Version, choose
“1.9.3-p392”. For Spree Version, choose “1.3.x”. Add the git url for
your repository like:
git://github.com/username/store-1-3. Then click
Note that I’m using the read-only version of the git url. That’s because I never edit code on the production or staging servers and never write from them to the git repo. I don’t store git credentials on a production server anyway, so I wouldn’t be able to.
The Deployment Service and Ruby Versions
Note that I’m specifying ruby 1.9.3 because although 2.0 is strongly preferable, the deployment service’s puppet scripts are not compatible with 2.0 as of this writing. If you select ruby 2.0, your install will fail.
That gets us to the point where the store parameters are known but the server parameters need to be specified. Under the Add Server section, put in a fully-qualified domain name such as “store.mydomain.com” (use your own domain).
For IP, just use “127.0.0.1”. Select the “App Server” and “DB Server” roles, then select “2” for the number of unicorn workers. Click Add.
When you configure the real server, you’ll want to set the number of unicorn workers to the number of cores on your production server or “2”, whichever is greater.
Since that’s the only server we’ll need, go back up and select “Complete” for Configuration Status, then click Update.
At this point you’ll need to wait a minute for the deployment service to generate your configuration. Take a moment to check that vagrant finished configuring your server. You’ll see an error about the window system drivers failing to install, that’s fine because this is a server with no window system.
When you’re ready, run:
If there is any other standard software you like to have installed on your machines, this is a good time to install it. This box will be packaged and redistributable, so get it in there now.
I’ve got a gist of software install commands I like to use for my customized boxes.
SSH Configuration on Non-Vagrant Ubuntu
The vagrant box will already have reverse DNS lookup on ssh connections
disabled, but regular Ubuntu won’t. If you’re using regular Ubuntu, edit
/etc/ssh/sshd_config and add the following line at the end:
Save and exit, then run:
Add the line (or see spree sudoers gist):
spree ALL=NOPASSWD: /usr/local/bin/bluepill
Save and exit.
This will give you a root login session, which is what the deployment service needs.
Now go back to the spree site and refresh the page. At this point you should see an Initialize Configuration command under your Server section. Cut and paste this command into the vagrant box, then go get a cup of coffee/tea/etc. Scratch that, go get a meal. This is going to take a while.
You may see some error messages go by while the machine is installing. That’s normal. Unless it crashes with an error, don’t worry about it.
I don’t trust my production system, so I never use it to log in anywhere else on the network. Therefore I don’t usually generate ssh keys on it, although I will add public keys to the authorized list so I can use ssh keys to log into the production machine. Of course, with vagrant you don’t need to worry about that since it already has the vagrant credentials builtin.
I also use password authentication rather than keys to do my Capistrano deployments but you may want to use ssh keys for that purpose. I’m not going to go into that here. If you use passwords with ssh rather than keys, make sure you have some other security mechanism to protect your server from the outside world. For example, make sure that only whitelisted IPs (or no IPs) from outside your network can reach the ssh port on your server. Passwords are not a safe way of securing your ssh server.
There’s probably quite a bit to be said about mysql security as well which I’m not much of an expert on. I make sure that there’s only local machine access to my database and that all of the accounts have passwords. I’ve left root without a password to simplify the instructions here, but take care to secure your mysql instance better than that.
When the install is finished (errors and all), go back to the spree site and visit the deployment page for your store. Refresh the page if you have to.
There will now be a new command available under Update Configuration. Copy and paste it into the vagrant box, replacing the password with what you’d like the database password to be. I’ll set mine to “spree”.
Exit the root shell.
The Rest of the Base Box
Remove the domain so there is only a hostname in the file.
Change the entry for “127.0.1.1” to “store” instead of “precise64”.
Change to a password of your choice, such as “spree”. Note that you’ll need a better password than that for a real system, but for vagrant it’ll do.
There are two accounts named “spree”:
- the MySQL database account
- the Linux account
We just set the Linux account password. We’ll be setting the MySQL password with a SQL statement below.
I make them match here, but they can be set separately.
The MySQL password needs to match the one found in
/data/spree/shared/config/database.yml, which was set by the
FACTER_DB_PASS part of the prior
Change the UMASK line to:
Save and exit. Log off and
vagrant ssh back in.
Copy the “production” section in its entirety to the bottom of the file and rename it “staging”, then save and exit.
Paste in the following (or see [secret_token.rb gist]):
Save and exit.
Remember when we initially created the application, and I just called it
store, then renamed the folder to
store-1-3? This is why. The
application name is generated from the name used when you created the
app. It takes your name and CamelCases it, removing punctuation. So
Store. It’s easier when upgrading when your app name
doesn’t change with the version, so that’s the name we want to use.
Paste (or see devise.rb gist):
Note that while spree 1.3 does not need the
devise.rb file, I’ve set
you up so that you’re ready for 2.1 (which does) without having to
further modify these files.
These machines are just for test purposes, so I’m using secrets which obviously aren’t very secret. For a real production machine, you’d want to generate a random secret for each of these files separately, using the command:
You’d run this command in your development directory on the dev machine, then copy and paste into each file, generating a separate secret for each file.
Also note that the “secret_token” is being deprecated in favor of “secret_key_base” in Rails 4 (which spree 2.1 is based on), but secret_token still works. I’ll be using secret_token exclusively in this post since it’s backwards compatible and these vagrant boxes don’t need production security.
See http://guides.rubyonrails.org/upgrading_ruby_on_rails.html for more information.
Change the “app.gid” line to:
Save and exit.
Groups and Setgid
On a production machine, you shouldn’t be sharing an account to administer it, such as the spree account. In order to have proper accounting of who is doing what on the server, accounts should never be shared.
As it happens, vagrant doesn’t want to log in as the spree user anyway, so it’s best just to establish good practices and make it so you can administer spree properly from the vagrant account anyway. While you could sudo to the spree account all of the time, it’s more convenient to allow your admins to work in the spree folder correctly without sudo.
We’ve added the vagrant user to the “www-data” and “spree” groups, which
gives it write access into the
/data/spree folder tree. We also need
to make the converse true so that the spree user can use the files
written by the vagrant account.
We’ve changed the default umask so that files we write will allow the
group to have control. The
find command above also sets the setgid bit
on all of the directories in the tree so that any files written to the
tree automatically get the shared group assigned, not the user’s default
Finally, we need the spree process on the machine to run with the group
id, rather than group “spree”. That’s what changing “app.gid” in the
master.pill.erb file does.
This setup saves you a multitude of small headaches.
The Vagrant Package
Since this vagrant box will more often be used for staging than for production, we’ll make another change before packaging it up for reuse.
Edit these two files and change “production” to “staging”.
Now logout of the vagrant box.
Now we’ve got our “fake” production machine configured and ready to go. Since we’re going to be setting up a staging machine based on the same configuration, let’s take a moment to save our work so we can just clone this box rather than go through the whole setup again.
On your dev machine:
This will shut down the vm and export it to a file called
You can then save it in the box archive and give it a name, then switch your current box to it:
If you’re going to distribute this box to other machines or over the net, then move and rename package.box to somewhere outside this directory. Otherwise:
Comment out the line (or see initial Vagrantfile gist):
Before the final “end” in the file, add the lines :
Save and exit.
sudo vim to change “store” to “production-1-3” in these two files:
Change “staging” to “production” in the files (use sudo on the second):
Now logout of the vagrant box.
Dropbox and Vagrant
If you’re like me, you work in multiple locations on multiple computers and use dropbox to pick up where you left off. Vagrant poses a special challenge here, since vagrant needs to agree on machine identifiers with the installed virtualbox. Virtualbox will be different on the two machines. While this is a good thing since it’s difficult to synchronize virtual machines with dropbox, it’s nice to be able to use vagrant seamlessly between the two.
If you bring up your vagrant box on one machine, it will note the
machine identifier in a file in the
.vagrant directory. Running it on
another machine with dropbox will confuse vagrant and it will always
overwrite this file with the current machine’s id, which causes
headaches between the two machines.
The simple solution is to force the two machines to agree on an id. Once you’ve brought the vagrant box up on your first machine, issue the following command:
Now, bring up the vagrant box on your second machine (say, when you go
home). Once the machine is up,
vagrant halt it. Now:
This will tell vagrant to use the same id as on your other machine.
Now make sure there is no virtualbox gui running. Also make sure you did
vagrant halt above.
There are two more files you need to edit to complete this. The first is:
Find the machine with the id from the
cat id command above, then copy
the new machine’s id in its place (from
Also, note the
src directory for your machine, since you’ll be going
Save and exit. Now:
Again, replace the machine id. Don’t mix this up for the hard disk id.
It’s in the
<Machine> tag, not under it.
Now your machines will both allow you to use the vagrant box and coexist peacefully. If they ever get out of whack, shut them down and copy the id from the id file.
SSH and localhost
One of the things we’re going to be doing a lot of is ssh’ing to the
local machine, since virtualbox and vagrant default to NAT’ing your
boxes through the local ip. This is fine, but ssh is paranoid about the
IP address and actual machine matching up each time you connect. Since
you’re going to use the same IP to connect to multiple boxes, ssh is
going to complain quite a bit if we don’t work around that. I’ve added
the following to my
~/.ssh/config file, which seems to smooth over
ssh’s ruffled feathers without sacrificing security:
Host 127.0.* localhost StrictHostKeyChecking no UserKnownHostsFile /dev/null
This needs to go at the top of the file, in my experience.
The most important thing to know about capistrano is that it allows you to control your deployment from your development machine. I had no introduction to capistrano before using the deployment service, so I thought it was something you used while logged into the production machine.
Capistrano uses ssh to connect to the target machine and then does all of the steps you would normally need to do by hand, including updating the git repository, compiling assets and so on.
It additionally maintains separate working copies for each release you do. This is important, because if there were a single working copy from git, any local file modifications would be lost when moving back and forth between releases. Capistrano avoids this issue by keeping each release in a separate directory.
One additional thing to know is that there are at least two situations which capistrano treats in a special way; the initial deployment, which is called a “cold” deploy, and when you have new migrations to run. Both require an extra keyword on the capistrano command to handle those situations. We’ll see that when the time comes.
The deployment service provides you with the capistrano recipe, which gets added to the project directory on the dev machine. We’ll configure it to manage a production server first and when that’s working we’ll add a staging machine into the mix.
We’ve already added capistrano to the
Gemfile earlier, now run the
following command to enable it in your project:
This will create a couple files, one of which we will replace with a recipe from the deployment service.
Go back to the spree commerce website and go to the deployment service page for the store. Underneath the Environment Details section, under Git url for your application, there is a link to a capistrano recipe.
Click the link, then copy and replace the contents of the file
config/deploy.rb in your project (or skip the following edits and use
my initial capistrano recipe gist). If you use my gist, change the
repository url to include your name.
Edit the file and change the following lines to include the vagrant port number “2222”:
Also add the following lines to the
namespace deploy do section:
Simple Capistrano Mistakes
Capistrano will only deploy changes which have already been pushed to the repository url supplied to the deployment service. It’s an easy mistake to commit development changes and forget to push them to the network repository. You’ll end up wondering why they don’t show up when you deploy to production, while capistrano can’t fetch anything not in the network repo.
Another thing to remember is that the branch matters. Capistrano will only pull the configured branch. So if you’ve been working on a feature branch, remember that you will need to merge or rebase those changes onto the branch configured in capistrano in order for those to make it to production (the “master” branch in our example here).
Let’s commit our changes:
Finally, do a cold deploy:
You’ll be prompted for the spree user’s password (“spree” here). Enter it and go get a cup of tea.
If everything goes as planned, you now have a production server! If you point your browser at http://localhost:8081/ you’ll be able to see it.
Sometimes the startup for bluepill reports failed sometimes when you deploy, complaining about spree.sock not existing or some other error. In my experience it’s usually succeeded anyway in this case, at least, so long as it’s bluepill throwing the error and not some earlier part of the deploy.
If spree hasn’t restarted, a manual restart will do the trick.
Watch the log for the unicorn workers to restart. You’ll have to enter the tail command pretty quickly after the restart, but spree takes enough time to come up that you should be able to catch it.
Ctrl-C will stop tail and you can log off the vagrant box.
Important Capistrano Commands
cap deploy- deploy to the default environment (w/o migrations)
cap production deploy- deploy to the environment named “production”
cap deploy:cold- deploy for the first time (migrations, starts instead of restarts)
cap deploy:migrations- deploy and run pending migrations
cap deploy:rollback- roll back the last deployment
Production and SSL
Note that at this point, since the vagrant box thinks it’s a production server, it will require ssl for checkout and the admin page. Since we don’t have ssl forwarded to the machine, you won’t be able to view either of those.
Unfortunately the view won’t be very interesting, since there won’t be anything in the store to sell, however. Which brings us to…
The point of this post is to help with migrating a real store, but we’re going to just set up sample data here. We’ll migrate the sample data to the new store. Since this store has no extensions or other complications, you may have a harder time with a real store. I’ll try to note the issues I’ve had with mine and how I dealt with them.
For now, generate some sample data:
Log off the box when it’s done.
You can check the store and see that it’s full of Spree products now. Yay!
Before we’re done, we’ll need to shut down this box.
Important Rake Commands
bundle exec rake db:bootstrap- now deprecated, this was a spree-specific rake task which went through the full database setup, with sample data. See below for how to do this with two commands now.
bundle exec rake db:reset- runs rake db:drop, db:setup
bundle exec rake db:[create|drop]- create or drop the schema, but I find the mysql command line to be faster if you’re running just this one command
bundle exec rake db:setup- runs rake db:create, db:schema:load and db:seed
bundle exec rake db:schema:load- load the schema from schema.rb, faster than running migrations when you’re starting from scratch
bundle exec rake db:seed- load the basic seed data that all spree instances should have
bundle exec rake railties:install:migrations- import the latest migrations from your updated gems into your app
bundle exec rake db:migrate- run the latest migrations for your database
bundle exec rake db:migrate:rollback- undo the last migration - this is not always possible if the migration was not reversible
Here’s a StackOverflow post on rake commands going over more of the commands in detail.
Recovering from Database Screwups
If you’re working with sample data, it’s easy to erase mistakes and start over with a clean database.
I’ve occasionally run into an “unexpected token” error when reloading
sample data. The solution is to
rm -rf tmp/cache and run both of the
above commands again.
Configuring the Project for Staging
We could move on to version 2.1 at this point, but if you’re serious about deploying an upgrade, you’re going to want to run through the entire process, including deployment, without taking down your real production server. That’s what staging is for.
We’ll need two things. The first is a staging environment file.
Change the line with local requests to true (or use this 1.3 staging.rb gist):
This allows error messages to be displayed in the browser. For pretty much all other settings, we want this file to match production since we’re using staging to test the production deployment process.
Save and exit.
The other thing we want to do is make it so we can go through everything on the staging server, including the checkout and admin interface.
Unfortunately, that requires using ssl, which is a pain to set up with vagrant. Instead, there’s another way. We’ll tell the project not to use ssl in staging.
Find the file
config/initializers/spree.rb and modify the section (or
use this spree.rb gist):
Save and exit. Now:
Configuring Capistrano for Staging
The other thing we’ll need is a capistrano staging configuration.
Remove the lines (or use this capistrano recipe gist, changing repo username):
After the first two lines in the file add:
Save and exit.
Now add (or use this production.rb gist):
Notice that the port number is different than the one we deleted from
Save and exit. Now:
Add (or use this staging.rb gist):
This tells capistrano to address two different servers, one for staging and one for production. Since presumably you’ll be deploying to staging more frequently than production (you are testing and working through issues, right?), staging is set as the default target for capistrano.
cap deploy will push the staging branch to the staging server.
You don’t need to specify anything else to target that server.
The deployment process will also deploy a different branch than master, the one we’ve used so far. This is so you can separate your work-in-progress from what’s pushed to the production server. It also allows the staging server to have a slightly different configuration than production so we can customize it if necessary.
When pushing to production, we’ll now need to specify that as the target
with the command
cap production deploy. Everything else will work the
I’ve assigned the default vagrant ssh port, 2222, to the staging server. That’s because normally, you’ll be running the staging server in vagrant and using it frequently. Running production won’t happen in vagrant in the real world, so I’ve given it vagrant’s secondary ssh port, 2200. When you run a real production machine, you’ll just use the machine’s ip address instead of 127.0.0.1.
Because vagrant assigns the ssh port dynamically, this means that you’ll have to start the staging server before you start the production server. You won’t want to start the production server without the staging server running, otherwise it will be assigned port 2222 instead of 2200.
I know this is fragile and I don’t like it, but I haven’t found a simpler way to fix this.
In my experience, the setup I’ve provided in the Vagrantfile does that
automatically when you just run
vagrant up. However, YMMV, and it
doesn’t hurt to pay attention to this step. If you get them backwards
for some reason, deploys will fail and you should be able to tell why.
If you’re in doubt, vagrant ssh to the machine and check which
environment is being called in
Let’s add these files to the repository:
Configuring Vagrant for Staging
In order to test our staging server, we’ll need to create the server and
start it up. Note that we turned off the production machine in the last
vagrant halt, which is important.
Since we already have a Vagrantfile in the current folder, let’s make it work for both machines:
Add the lines (or use this Vagrantfile gist):
Save and exit, then commit the new Vagrantfile:
Let’s also set up a staging branch for our work with the staging server:
Git Workflow with Staging
There are many ways to do git workflow.
I’m going to do all of my work on the staging branch, which is the only branch which will push to the staging server.
Once I’m happy with the commits on that branch, I’ll merge them to the master branch, which pushes to production:
You may prefer a different workflow such as rebasing. I don’t think that will work if it removes the staging branch in the process, but I can’t really speak to that.
One advantage of not working on the master branch directly is that you can be interrupted while working on staging to fix some pressing issue on master. Just commit your staging branch without merging to master, go work on master (or branch from master and merge) and deploy, then switch back to staging.
sudo vim to change “store” to “staging-1-3” in these two files:
Log off the staging box, then:
Once that’s done, seed the database:
Logout of the vagrant box.
Verify by going to http://localhost:8080/. You should also be able to visit http://localhost:8080/admin. Username is “email@example.com” and password is “spree123”.
Leave the staging and production vagrant boxes running, which will be important for the next step.
You can learn more by visiting the vagrant multimachine page. The
important thing to know is that when you use the multimachine syntax in
your Vagrantfile (as we’ve done), all vagrant commands will take a
machine name after the normal command, such as
vagrant status staging.
Some commands will require a machine name, but ones that can operate on
both machines (such as
vagrant up) will do so if you don’t specify a
machine. Be conscious of which machine you want to be affecting.
Also, you may or may not want to run multiple machines at a time, given your system resources. These boxes are fairly light, but they still have some weight. Sometimes you’ll need to have more than one machine on at a time. If your system can’t handle it, you’ll want to reconsider your development environment.
In particular, don’t try to get fancy turning machines on and off differently than outlined here, since the ssh ports will likely get confused and you’ll end up with errors. The safe approach is to get the machines up then not mess with them.
Now that we’ve got our old store set up, let’s get our new one.
On the dev machine:
Paste (or see devise.rb gist):
Save and exit. Now:
This looks mostly like it did for the 1.3 store (or see 2.1 Gemfile gist):
Find the unicorn line and change to:
Find the capistrano line and change to:
Also add these lines at the bottom:
Save and exit. Now:
Now we’re going to update to the “2-1-stable” branch instead of the published gem. The team is constantly fixing issues with spree, and many times I’ve had to address an issue by updating to the latest changes in the stable branch. I would have done this to the 1.3 store as well if it had been a real store.
Replace the spree gem line with (or use the final 2.1 Gemfile gist):
Save and exit. Now:
Visit http://localhost:3000/ to test the store. Ctrl-C stops the server.
Updating the Bundle
When updating spree to the latest commit in the branch, the above set of commands will become familiar. We’ve just done a mini-upgrade of spree, from the 2.1 release to 2.1-stable latest. Those commands are a staple of maintenance.
One thing to be aware of is the difference between
bundle update and
bundle update spree. You want to use the latter usually, not the
bundle update revisits every gem in the Gemfile, upgrading to
the latest if possible. There’s a much bigger risk of things breaking if
you update every possible gem, especially when you know most things are
working and you don’t need to.
bundle update spree only updates the
spree gem and its dependencies, reducing the risk of change.
bundle update here since this is a new install and we want the
latest of everything as our starting point. Since this store’s never
been deployed, we don’t know a priori that earlier gems will be any
better than the latest in any case, unlike with known working code.
We’ll do the exact same instructions from the “Setting up Git” section above:
Add the following lines to your
.gitignore (or same .gitignore
Save and exit. Next run:
You’ll also want to set up a
store-2-1 repo on github. Go do that then
Use your own username instead of “username” of course.
Staging for 2.1
We’re going to perform the migration and put it on a staging server. We’ll skip using a production server, since the process will be exactly the same as upgrading to the staging server. We use a staging server to make sure the entire process works flawlessly prior to upgrading the store, the same store which is actually in production and running our business.
Copy the following files from
Vagrantfile, edit the port forwarding to forward port 80 to port
8082 for the staging machine (or use the final Vagrantfile gist). Also
comment out production since we won’t need it at the moment.
When you’re done with the migration, you may want to change this back to 8080 when you don’t need the 1.3 machines any more.
config/deploy.rb, edit the repository url to point to
git://github.com/username/store-2-1 (or see the final capistrano
recipe gist). Also add the line:
to the deployment section with the other lines generating symlinks.
config/deploy/staging.rb, edit the port to be 2201 (or see the
final staging.rb gist). This is the port assigned by vagrant to the
third machine brought up. That means that the 1.3 staging and production
machines will need to be up first. When you’re done with the two 1.3
machines after the migration, you’ll want to change this back to 2222.
Change the local requests line to (or see the 2.1 staging.rb gist):
Save and exit. Now:
sudo vim to change “store” to “staging-2-1” in both of these
Log off the vagrant box and run:
Verify that it has come up by visiting http://localhost:8082/.
Look at everything we’ve accomplished so far. Seems like a lot. And yet we still haven’t even talked about the actual migration process.
I wish I could say this were the home stretch, but in reality, most of the work will probably go into figuring out what has changed out from under your app in the move from 1.3 to 2.1.
The standard upgrade path is described at http://guides.spreecommerce.com/developer/upgrades/index.html. There are two guides, one from 1.3 to 2.0 and the other from 2.0 to 2.1. I’ll go over the process for both in one step here. We’ll be working on sample data with no extensions, which will be fairly straightforward.
The harder part will be adapting your particular environment. I’ll go over the specifics of how I was affected and what strategies I used, but you’re somewhat on your own here.
Upgrading in Stages
I recommend refining your upgrade procedure in trial stages:
uncustomized, vanilla store (as in this post) with sample data and no extensions
vanilla store with production data
full upgrade with extensions, customization and production data
the actual deployment
Only after you’ve successfully gone through the first three stages will you upgrade your actual production store. Additionally, in each stage, you’ll be deploying to a staging store, which tests the deployment process as well as the upgrade process. It’s good to close the deployment circle early and often.
The reason I break it up like this is that I really need to have a level of comfort with the upgrade process itself, in a vacuum so to speak, before I start working on the inevitable issues I’ll come against dealing with extensions and my customizations. That’s because it sucks to spend time chasing bugs in your code when it’s actually a problem with the upgrade process, and vice-versa.
Let’s start off by upgrading our project directory on the dev machine.
We’ll start with the Gemfile. Fortunately, this is easy since we already have our 2.1 directory set up.
Let’s also create a temporary git branch for our work.
Now we need to update the Rails environment files and the application config. Here’s where having the same application name from version to version helps. If your application names aren’t the same, you’ll need to update it in these files:
Now let’s get rid of a migration which happens to be a duplicate from 1.3 to 2.1:
In the following two sets of files, remove the lines which refer to “spree_promo”.
In the first set, also change “spree_core” to “spree_frontend”:
In the second set, also change “spree_core” to “spree_backend”:
Finally, let’s do the update to 2.1:
That’s it. Test the new store version at http://localhost:3000/.
Now commit our changes so we have a checkpoint:
Missing Template “spree_application” Error
For the longest time I couldn’t figure out why I’d get this error sometimes on my dev machine when attempting the upgrade.
The instructions I give above avoid that error. The magic secret is:
For some reason, this directory is the source of all kinds of weird errors whenever it doesn’t like what you’ve done. It’s now my go-to first response to any kind of unexpected error, for sanity’s sake. Don’t worry, the directory regenerates itself.
A Trial Deployment with Sample Data
I really like to close the circle to deployment as quickly as possible so I know that the basic process is ok before adding complexity.
We have a staging machine ready to do this. While this won’t be wholly indicative of how the deployment will go with 2.1, since we’ll be using the 1.3 machine to test here and our 2.1 app has a whole different repository and schema, but it will tell us that the regular spree process is sound. It’s important to have a working baseline.
You’ll want the 1.3 staging machine to have a copy of the sample data. We’ll go through the instructions to generate that as we did on the fake production machine.
Log off the vagrant box.
You can see that the staging server now has the sample data at http://localhost:8080/.
Since we’re not going to be keeping this git branch, I’m going to do the upgrade without bringing our changes into the staging branch. That means we’ll need to change the deployment file and push the current branch:
Change the branch to:
Save and exit. Now:
If you get a bluepill error, ssh onto the vagrant box and
sudo bluepill spree restart, then
tail -f /data/spree/shared/log/unicorn.stderr.log. If you had the
bluepill error and haven’t done this, it will manifest as an unknown
column error “spree_products.count_on_hand” when you visit the
If everything works, you’ll be able to see your new version of the store at http://localhost:8080/ again. You should refresh your browser cache just in case even though rails should ensure that the assets are refreshed anyway. Congratulations, this is a major part of the upgrade accomplished!
If it doesn’t work, you’ll need to troubleshoot. Sorry, you’re on your own here, but the spree google group and irc channel on freenode are a great help in this case, as are google and stack overflow. You’ll want to bring your data onto your development machine and use mysql, which again are out of scope for my post here.
As I mentioned, if you’re not really into my “new store” approach, at this point you could just integrate the changes into staging and then production, upgrading your real store and being done with it. At least, if you don’t have any application customizations or extensions. You can follow my suggestions for those in the following sections, or follow along with my entire process if you choose.
Extensions and Customizations
Extensions are the more straightforward of the two, so I’ll start with those.
As I mentioned at the beginning of this post, do your homework with regards to 2.1 compatibility before you begin the upgrade process.
I won’t upgrade an extension here, but here are tips and strategies:
- Upgrade one at a time - I generally do the easy ones first and add the harder ones last
- Use the github branches - usually you won’t have a choice, but always choose a stable github branch for your extension’s 2.1-compatible version. Count on needing to update when fixes come out as they come out.
- Not all extensions have migrations - when they do, you’ll need to use the railties and migrations-related rake and cap tasks. When they don’t, the documentation will usually not have those in the upgrade instructions
- Don’t reload your database - you should be testing deployment to your staging machine each time you onboard an extension. You can keep operating on the data we set up the first time, you don’t need to reload it. If things get too messed up though, you can always reload it the same way we did it the first time, so long as you tell capistrano to perform migrations.
bundle update. Once you’ve gotten a spree version locked in, you don’t want it to change while you’re doing the rest of this work. Otherwise you’ll have to go back and retest everything you’ve done.
bundle installshould handle you for the most part when installing extensions. If there’s a gem conflict, try
bundle update [gem]on the new gem for resolution before resorting to a spree update.
Now for customizations. Whatever your app overrides within spree, count on having to check whether it still works. This is another reason I like the “new store” approach, it allows me to bring in my customizations one at a time, starting with the easy ones.
Deface and view overrides tend to work, for the most part. Deface will at least throw errors in the development log which you can search for. Still, make sure to visit every customized view and see it with your own eyes.
With Spree 2.1, the biggest change to deface is the specification of erb tags. In the past, the deface selectors would specify outputtable erb code with “code[erb-loud]”, for example. The latest version of deface upgrades them from “code” to “erb” tags, so that would become “erb[loud]”. I had to find and replace all of these in my overrides.
Whenever you do a model or controller override using class_eval, you’ll need to verify that the overridden function hasn’t changed in how it works. If it has, you’ll usually need to update your override to incorporate the changes from the overridden code, such as method calls and instance variables.
With Rails 4, the mass assignment through the attr_accessible syntax has been deprecated and extracted to a separate gem, in favor of controller params whitelisting. You can read more about it here: http://guides.rubyonrails.org/upgrading_ruby_on_rails.html#upgrading-from-rails-3-2-to-rails-4-0. You’ll either need to add the gem which provides attr_accessible or update your code.
In Rails 4, the “match” directive in the routes file has been deprecated. You’ll want to replace it with the appropriate “get” or “post” verb. This is also described in the Rails update guide above.
In spree 2.1, views which used to provide @current_user instead now use the method spree_current_user. You’ll want to examine your deface and view overrides for these references.
In spree 2.1, some controllers which used to provide @order instead now use the method current_order. Update accordingly.
Testing the Full Production Migration
If you’ve held on this long, you’re almost out of the woods.
So the tricky part here is to get the old database into the new one. I’m not talking about just dumping the old database onto the new machine, schema and all. I’m talking about allowing using a fresh 2.1 schema and getting all of the old data into that. This can be a hassle if you don’t have all of the extensions done right, or if you’ve removed extensions in the past without removing their data from the database. It can be like trying to fit two legs into one-leg pants.
There’s also metadata about the schema in the database itself, which won’t be the same as the metadata from the old database. You’ll want to exclude this data from the transfer. It basically boils down to not bringing over the list of applied migrations, since the new schema already knows the applied migrations and the lists may not be the same.
You’ll want to perform this process twice, once to test the migration and deployment processes, and once for the actual final migration. Usually I’m building a new server with the new version and then switching over to it rather than upgrading my old server in place, but either approach works. Whichever approach you take, when you’re doing the production migration, you’ll need to take down your store site from the outside world while the data is being migrated so that no new data makes it into the old store.
The practice run will be done with the staging box, while the production upgrade will be done however you choose, to a new server or to the old one. The only thing to note about upgrading the old server in place is that you’ll need to hold the old and new schemas in mysql at the same time, which means they’ll need different schema names and you’ll want to update your database.yml file to point at the correct schema when you switch to the new one. I won’t go into that here.
Let’s get the database from the production machine. You’ll be doing this on a real production box, but since I’ve only got the fake production box here, these instructions will look the same as when I did it for the test migration. Remember that you’re going to be using a real machine, so you’ll have to adapt my instructions here to your environment.
We’ll also get a copy of the uploaded assets directory, although we won’t need this until later.
Copy the sql file to your dev machine in the store-1-3 directory, then log off the box.
Now we’ll put it on the 1.3 staging server for the migration process. If you were using mysql in development, you could just as easily do this on your dev machine, but I don’t have mysql on this one. Fortunately, the staging machine is fine for this purpose. All we need it to do is run the migrations for us.
Now we just need to run the migrations. Since we’ve already deployed the 2.1 upgrade to the 1.3 machine, all we need to do is to tell it to migrate the database:
Finally, we’re going to export the migrated data to be imported into the clean 2.1 store. We’re going to use the mysqldump command again, but this time we’re going to tell it to leave out the schema_migrations table. We’re also going to tell it not to include table drop/create instructions, since the table schemas we want are already created in the new database. Finally, we’re going to tell it to include the column names in the dump since the default insert command is based on column ordering, which may also have changed.
Important Mysqldump Parameters
-t- don’t include table drop and creation commands
-c- include column names instead of inserting by position
--ignore-table=[database].[table]- don’t include the specified table in the dump
Log off the vagrant box. Now:
If everything goes well, you should be able to see the clean, upgraded store at http://localhost:8081/.
If not, you’ve probably got some data in your database from tables which no longer exist in your new version. If you get complaints about tables not existing, you’ll have to edit them out of the sql file and reimport. Remember what you’ve done because you’ll need to do it all again when you upgrade the store for real.
You’ll also need to clear out the 2.1 staging server’s database before importing again. You can do that with the mysql schema commands.
There are also command line switches for mysql which would let you roll back the import if it were to encounter an error, in which case you wouldn’t have to clear the database out. Unfortunately I don’t recall what those switches were.
Once you’ve got it all working, spend some quality time with the staging store and really make sure everything is working. Go through some fake purchases. Pay attention to the details. This is your last chance to save yourself from mistakes you’ll have to fix in production.
Vagrant Shared Folders
By default, vagrant uses the virtualbox shared folders facility to mount
the Vagrantfile’s directory into the vagrant box at
I’m using it here to pass files between the boxes which you’d normally need something like scp or sshfs for.
While it works fine for these individual files, don’t try running your rails app out of the shared folder. Unless you configure shared folders for NFS, it’s fair to call their performance excruciatingly slow for large numbers of file accesses.
The Final Countdown
Now that we’ve seen everything working in staging, we’ve had plenty of practice getting things out to production. I won’t even go through the actual commands. Consider it your final exam. I’m sure you’ll do fine.
The things to remember are:
Close off access to your production machine prior to this process. Give yourself plenty of time, at least a couple hours, in case there are snags. Let your customers know when the site is expected to be back up, and how to contact you through other means in the meantime, if appropriate. Also notify your other store admins and close off their access as well. You don’t want anyone changing anything while you’re migrating.
If you’re moving to another server, don’t forget your ssl certs! I haven’t discussed that at all. In fact, I proxy from a separate machine, so even when I add a new spree server, I just repoint the proxy without having to mess with the certs. I don’t necessarily recommend that setup, but it’s handy in this case.
Bring both the database contents and the uploads directory to the new machine after you’ve closed off the old store from the outside. You’ll need the uploads just as much as the data.
Have fun with it. Just kidding, this process is undoubtedly tough, but what can you do? Follow this post! It makes it easier, I swear. :)
Also, if you use the vagrant box for staging which I’ve outlined here,
don’t forget to set the application name to match yours in the box’s
/data/spree/shared/config/secret_token.rb file, and to use actual
secrets rather than my slugs.
I meant to write some other stuff as well, but this should keep you busy enough I think. I’d love feedback and will try to incorporate good suggestions into the post. Thanks for reading!