Automating acceptance tests is a very worthy and lofty goal. Doing so gives a team and a business an ability to continuously ensure that a software application does what it is supposed to do. However, it can be rather challenging to implement, sometimes to the point that the challenges seem to outweigh the benefits. With the right tools, the right design and the right approach, though, getting the benefits while minimizing the challenges can be done. In particular, Angular, Protractor and ASP.NET 5 provide a means to achieve this when building web applications.
Acceptance tests verify that the requirements of the system have been met. When building software from requirements defined by user stories, acceptance criteria provide a clear and concise definition of what the software should do; they form the common measure used by the team and the business to determine if the behavior is correct. Acceptance tests, then, are based on acceptance criteria. Other tests should verify small details and edge conditions and ensure that the pieces fit together correctly, but acceptance tests verify that the complete, running system does what it should, from an end-user perspective. Automating these to run regularly ensures that new functionality works correctly and that existing functionality continues to behave as expected as the system evolves. This removes the need for extensive, manual regression testing in order to have confidence in a release, enabling an organization to deliver valuable features more frequently, with lower cost.
There are clear benefits to these user-focused, behavior-based tests but automation can be challenging. First, it typically requires some sort of UI automation, which can be time-consuming to script or build and result in brittle tests that are prone to break when the UI changes. Additionally, software often runs on a database in the back-end and may integrate with other systems, so achieving test automation requires an automated means to ensure that the data and related systems are in a known state, which again can be time-consuming and brittle. The time commitment can be discouraging for both the business and team. Brittle tests cause a lot of trouble, as they require time to investigate and fix, and can lead to a lack of confidence in the tests; eventually, people may begin to ignore failures as false negatives and actually let real problems slip through. The combination of time required and reliability issues (the two also being related) can deter automation.
It seems, though, that if we can find quick and reliable ways to automate the UI and control back-end state, we have the potential to build a very effective automated acceptance test suite with reasonable effort. The architecture of a REST API running on top of a database, with a JavaScript MVC front-end is pretty common these days. When implemented with ASP.NET 5 Web API and Angular, there are some particular benefits for test automation.
Angular has a dedicated end-to-end testing framework called Protractor. Protractor makes automating the UI in tests much easier, as it’s integrated with Angular and allows direct access to its elements. Interacting with page elements is lightweight, flexible and precise, even when working with dynamic content such as lists. With this capability, a test author can directly code the UI interaction as part of the test definition, in a way that sidesteps some of the challenges of UI automation and makes test execution reliable. The UI runs in an on-the-fly Selenium server during test execution, rather than needing to actually be deployed to a web server, which simplifies running tests. It’s also easy to run the same tests against multiple browsers, to ensure cross-browser support.
In the back-end, the new ASP.NET 5 provides some features that make test automation easier, with self-hosting and in-memory database support being the most interesting. Though there are tools and techniques available to achieve similar results with previous versions, the new, native features make this much easier.
Self-hosting can be achieved simply by a configuration setting in the Web API and a command line call to the .NET execution environment. This allows starting up the API as part of a build definition, without the need to have a configured IIS site and deployed code or a separate executable. Because the API can run without being deployed, the need for an additional dedicated environment (with a known and reliable configuration) is removed. With less moving parts and dependency, tests can be more reliable and faster to execute. Plus, this allows more tests to be run before any code is deployed, keeping development environments more stable.
Self-hosting is also useful in test scenarios which involve working with external systems that integrate via web services. A fake version of the external service can be created, which will be self-hosted and set to a known state as part of the test execution. Configuration of the Web API during tests can point it to this fake. There is an investment in building the fake service, but self-hosting allows simulating interaction without needing to actually deploy and maintain the fake on a web server. This approach enables focused and reliable test automation for applications that involve integration; it can have particular value when using a third-party service that has usage limitations or a pay-per-use model.
The new in-memory database option provides the ability to have a known and controllable state of the system’s data, without the overhead of managing an actual database instance. The architecture of ASP.NET 5 and the way in which the application is initialized at startup, combined with the in-memory database feature allow for substituting a fake database at a very low level. After that, all code follows normal production paths and behaves exactly the same as if there were an underlying database. There are a variety of options from there on how to initialize the state of the data during each test’s setup, in order to satisfy the initial conditions that the test relies on. This lightweight approach allows running data-dependent tests and checking their results without the need to maintain a test database.
The combination of features in Protractor and ASP.NET 5—with some minimal test infrastructure built around it—allow realizing the benefits of automating acceptance tests, while minimizing the difficulties. This provides a solution that runs fast, in-memory, with no external dependencies and can easily be wired-in to the automated build-test-deploy pipeline. The end result is a solid means by which to ensure software quality from an end-user perspective, which continues to provide benefit throughout the application’s lifespan. In a follow-up post, I’ll get into the details of how to put all of these pieces together.
About the author
