Having reliable testing in your development and deployment pipeline is essential when working in large and small projects alike, however recently I have started to run into issues with reliability when using karma-webpack to run my client side tests for a current project. The issue is that karma-webpack generates a bundle for every single test file, when the browser connects to karma it loads all of these test files in one go and also has to parse and execute them immediately, now on a powerful development machine and a modern browser this is no problem, but on your CI driven test targets you might start to run into issues when trying to load and parse hundreds of MBs of javascript in a reasonable time.
Now in my situation I have karma tests executing through WebDriver against 3 target t2.small instances running ie9/10/11, firefox and chrome, multiple branches can be running tests in parallel on these machines. I started to notice that the test runs would become less and less reliable with each new component that was tested with karma disconnecting from the target browser due to 10 seconds of inactivity.
It's pretty obvious when you look at the karma-webpack output and the DOM of karma when loaded in the browser that there is way too much javascript being loaded for a few hundred small tests. Here is just the first 10 files:
Asset Size
../test/client/components/SearchFilters/scaffold/FilterGroup.or.tests.js 2.88 MB
../test/client/blocks/Accordion.js 1.09 MB
../test/client/components/SearchResultsAside.js 3.93 MB
../test/client/components/Auth.js 3.79 MB
../test/client/components/PaymentIframe.js 3.77 MB
../test/client/forms/ValidationElement.js 3.76 MB
../test/client/forms/Validation.js 3.78 MB
../test/client/data/RateLoader.js 3.72 MB
../test/client/components/CountryOfResidenceSelector.js 3.31 MB
../test/client/components/SearchResults.js 3.62 MB
My first thought was that a lot of the code in these bundles is exactly the same, sounds like a job for the Commons Chunk Plugin. In you app you might use this plugin to generate a separate vendor file for all your node_modules dependencies meaning that when webpack recompiles your changes it only has to recompile your own code, so sounds perfect for reducing the size of the test bundles. I soon came across this issue though, karma passes the preprocessor a list of files and expects an output file for every one of those input files, so basically all those large bundles are unfortunately required.
Thought number two was that karma must have some configuration option to load the files in one at a time or in batches, waiting until the tests are executed to allow the next tests to load. Scouring the karma configuration documentation I found that this was not the case and it's easy to see how it wouldn't be viable with framework methods such as mochas (describe/it).only.
Thought number three was born through the karma-webpack readme and its "Alternate Usage" information. The simple solution is you have one file and you require every test spec file from within that single file, GREAT! Except not great because who wants to keep that file updated when you have hundreds of test files?
Generating the test files
So the simplest thing to do would be to generate a test file (or multiple test files) that can then be loaded by karma instead. Ideally this would be done by karma, but as we've already found out you can't provide a reduced file list to karma from a preprocessor.
So to implement my idea I grabbed a couple of dependencies we were already using in our build and threw together a simple solution for generating these test files, you'll find the code below. I use bluebird to keep the code easier to reason.
const path = require('path');
const Promise = require('bluebird');
const readdir = Promise.promisify(require('fs').readdir);
const stat = Promise.promisify(require('fs').stat);
const writeFile = Promise.promisify(require('fs').writeFile);
const recurse = Promise.promisify(require('recursive-readdir'));
const backSlashRegex = /\\{1,2}/g;
// A list of files that should always be included in the test files
// these will be included before the tests so you can register setup and
// teardown functions
const includes = [
'../setup.js',
];
function getPath(relative) {
return path.resolve(__dirname, relative);
}
function getRelativeFilePath(dir, file) {
if (file.startsWith('.')) {
return file;
}
// replace backslash with forward slash when using windows systems
return `./${path.relative(dir, file).replace(backSlashRegex, '/')}`;
}
function getRequires(dir, paths) {
// Generate a list of require calls
return paths.map(item => `require('${getRelativeFilePath(dir, item)}');`)
.join(' ');
}
function writeTestFile(relativeDir, files) {
const dir = getPath(relativeDir);
// The name of the file to generate, this is passed to karma
// This should also be added to .gitignore
const writeFilePath = path.join(dir, 'generated_test.js');
// group the tests in a mocha describe
const content =
`/* eslint-disable */
describe('${relativeDir}', function tests() {
${getRequires(dir, includes)}
${getRequires(dir, files)}
});
/* eslint-enable */`;
console.log(`Writing test file - ${writeFilePath}`);
return writeFile(writeFilePath, content);
}
function build() {
// Read the test directory and find all child directories
return readdir(__dirname)
.filter(item => {
// Skip any directories that don't contain test files
// this could be done with a glob or regex for more complex setups
if (item === 'helpers') {
return false;
}
// Check if the path is a directory
return stat(getPath(item))
.then(file => file.isDirectory());
})
.then(items => {
// Build up a map of promises for lists of javascript files
// Whatever pattern test files take in the system could be used here
const dirs = {};
items.forEach(item => {
dirs[item] = recurse(getPath(item), ['!*.js']);
});
// resolve the file path lists
return Promise.props(dirs);
})
.then(dirs => {
// Asynchronously write the test files
// This could also be done using map with concurrency to increase speed
const keys = Object.keys(dirs);
return Promise.each(keys, item => writeTestFile(item, dirs[item]));
});
}
module.exports = build().tap(files => console.log(`${files.length} test files created`));
Now clearly this was written for a very specific purpose in a very specific codebase, but with some minor changes it could be useful generically, it could even be made recursive to support batching files in subdirectories of subdirectories etc.
To fit this into your test flow all you have to do is add it to your test script in your package.json like:
"build-tests": "node test/build.js"
"test": "npm run build-tests && node ./node_modules/karma/bin/karma start karma/karma.local.conf.js"
And change your karma config to look only for the generated files
{
// ...
files: '../test/**/generated_test.js'
// ...
}
When using watch mode (no-single-run) you just have to remember that any new files won't be added to the tests unless you rerun the function to generate them, but you won't have to restart the watch at least.
Since using this solution random failures in CI have dropped to almost nothing (IE9 always loves being a pain), so I hope it can be helpful for someone else who has my same issues.