Development Journal: Unit Testing the Entire Website

I’m working on writing a unit testing framework for the entire Bangor Daily News family of websites, and it’s something that I’m struggling with. For that reason I’ve decided to journal my frustrations and the steps in the hopes that I can remember what happened on the other side of this.

What is Unit Testing?

Unit testing is basically writing code that checks other code to make sure that the code it is testing performs as expected and intended. Unit testing is something that every large scale project should be doing, and something that we as a company hadn’t been doing. That made it my problem.

May 27, 2015

The goal of the project is pretty straight forward: WordPress has a rather robust unit testing setup for their core files, and there are examples for how to unit test Plugins and Theme files. I’m going to combine all of these tests, install the website files over the WordPress core, and run unit testing on all of the files.

Right from the start I’m realizing it’s not as simple as copy pasta over core. Our site has heavily customized things, and we’re using many plugins and drop-ins that the Core unit tests don’t account for, so I’ve forked the Unit test framework, and I’m making the required changes.

Issue #1: Database

Our website is a Multisite install, which the core files are setup to handle, but unfortunately the default setup isn’t working. Connection is refused, the unit tests aren’t working, etc. To see what the problem is, I’m going to run a simple test.

Steps:

  1. Clone the WordPress development files.
  2. Create a fresh database table.
  3. Update wp-tests-config.php with corresponding DB info.
  4. Run phpunit

If the issue really is the difference between our website files and WP core files, PHP Unit should run without issue.

Result:

After doing the steps above, the result was a clean running unit test. That tells me that our database isn’t working with the WordPress core unit test setup. I’m going to test this further with multisite setup, but I think I’m on the right track.

WordPress let’s you create custom database error handlers, so I created one to print a backtrace for me whenever the DB isn’t working, and low and behold it worked and let me know that HyperDB, which we use because multisite, isn’t loading just the way it should, for one reason or other.

We heavily customized our config file, I mean a lot, so I’m taking a play from WordPress’ handbook here and creating a config file strictly for our unit testing. This means I’ll be cutting out a lot of the clutter, and the things that aren’t being tested currently, and hopefully that’ll get us to the point we can at least run the tests. I don’t care if they error out, I just want a fully running PHP Unit.

Because I really don’t care about HyperDB, and it’s just causing me heartache, I got around the errors by just not using HyperDB during unit testing. This is obviously a hacky way to handle it, but once I have a working unit test setup, I can work on forking HyperDB and fixing it’s implementation.

Next up to ruin my day is Batcache! That said, I also don’t really care about cacheing when I’m running unit tests, so I’m just going to shut that off for this as well.

Shut off caching and it found new errors. Sunrise was causing errors, but being wise to their tricks, I created a blank sunrise file. There were a few more errors, but I realized during the process that I was running Master tests against the 4.1 branch. So I switched to the 4.1 branch tests and ran again. PHP Unit started running.

It’s randomly throwing errors, but that’s fine. I track each error down and either skip it, or fix it.

June 2nd, 2015

The last time I ducked into it we were getting a lot of terminal issues where a rogue UTF-8 character was changing the character structure of the terminal. I spent a few days trying to figure out how to remove the data sets from the PHPUnit debug output. Figured that out today, so now I’m back tracking down individual tests.

After skipping all of the tests that caused PHP errors, or took more than a few seconds to complete, I finally reached the point where the tests would finish and the PHP Unit report would display. This is good. It’s filled with fails, but it completes.

Now I’m working on a BASH Script to automate the setup and tear down of the main repository code, so I don’t have to keep a duplicate copy of our codebase in the testing repository.

June 17th, 2015

I finished this a week or two ago, shortly after my last entry, but forgot to update. Basically what I ended up doing is looking at each fail, and if it was something that would take more than a couple minutes or a line of code change, I skipped the test, and left myself a note in the code as to why. I now have a passing Unit Test.

I’m going to publish this as this entire brain dump, and revisit the topic in a future post where I’ll explain the whys and the whats a lot better. For now, this is what you get.