Hi all,
I’d like to hear what others who are delivering complex non-blog solutions to clients with WordPress as a platform what they are using for automated Regression Testing?
For those not familiar with the term “regression testing” Wikipedia defines it as:
Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.
More telling Wikipedia says the following which is exactly what I’m experiencing on a project right now:
Experience has shown that as software is fixed, emergence of new and/or reemergence of old faults is quite common. Sometimes reemergence occurs because a fix gets lost through poor revision control practices (or simple human error in revision control). Often, a fix for a problem will be “fragile” in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a software bug in another area. Finally, it is often the case that when some feature is redesigned, some of the same mistakes that were made in the original implementation of the feature were made in the redesign.
With the global nature of actions and filters I’m finding that complexity starts ramping as I add more client-requested functionality and it becomes hard to get a complex plugin stable, especially if it uses a lot of calls to WP_Query and updates the database a lot.
The solution in my mind would be to set up regression testing with a series of “test cases” to comprise a “test suite.” In concept it’s not that hard when you are testing the HTML output of HTTP GET requests. But it gets a little more complicated when you have to test things when logged in via the admin console and/or to test jQuery interactions.
I’m setting this up as a community wiki in hopes we can gather best practices here but I’m really anxious to hear processes if any other WordPress professionals are using.
PHPUnit would come to mind, if the WP test suite wasn’t so broken, and if WP had been designed and written in a way that it could actually be tested properly. 😉
More seriously, you can test your plugins all you want from their functional standpoint with unit tests and the like. The issue is that these tests won’t guarantee that they’ll catch subtle chances introduced by WP upgrades, let alone that they’ll continue to work once plugged into a customized WP install.
Among the colorful things I’ve seen happen:
A subtle change in the WP API affects your plugin’s functionality, e.g. the hook you’re on used to get a term id and it’s now getting a term taxonomy id. (Chances are good that your test terms conveniently have the same id for both).
A subtle change in the WP API results in your receiving a
WP_Errorobject instead of the previously expected value offalseas bad input.Your plugin is added from within the mu-plugins folder, resulting in a subtly different code flow.
Your plugin worked fine until memcached or some other persistent store got enabled.
Your plugin worked fine until the despised switch_to_blog() got called.
A plugin changes the hook which it resides on when called, and unknowingly interrupts it as a side effect.
A plugin (un?)knowingly messes around with your input or output data to the point where things look broken even though you’re not at fault.
I could expand the list on and on, but those would be the key items that broke my own plugins. The two items are arguably catchable with unit tests. The next two as well, if you’re patient enough, but I’d argue that WP should not change the way things work when they occur. No amount of testing will work around the buggy implementation of switch_to_blog(). And the last two are hopelessly untestable.
Oh, and… don’t even get me started on the onslaught of attachments, auto drafts, revisions, menu items and what not that end up stored in the posts table.
Good luck… 🙂
You should strongly consider Selenium.
It allows you to record actions (e.g. entering data into a form, clicking a link) and then you can perform assertions. It also integrates with PHPUnit. I highly recommend checking out the two minute demo.
Selenium is probably usable but I think that in modern times you will find Codeception to be better and easier to use. For the simplest visual regression testing, it even has an extension that will take screenshots and automatically compare them for you.
Of course, Codeception WebDriver tests can go further and perform functional regression tests. You can fill out and submit forms, click buttons and links on the site, execute any JS, etc. You can use a real-life browser like Firefox or Chrome in your tests, or you can perform headless testing with PhantomJS. That means that you can even run the WebDriver tests for your plugin as part of its build process on Travis CI if you want to.
There are even several WordPress-specific libraries to help you get started: