A better testing framework

I have been thinking about writing a better testing framework. There are a few reasons why I have embarked on this journey to write it. The first and foremost is the fact that I had already written something similar almost 7 years back in Python2. The second reason is that I am forced to use a framework written in such a poor way that no one can put their confidence on the same.

The framework, as termed by them, is nothing but a bunch of scripts which are poorly strung together using JSON files, init.py files and some command line arguments.

Drawbacks of the current framework

The biggest flaw of the current framework is that of protection against test initiation. The framework is divided into two parts essentially, a part of the framework resides in the system where test would be run and the other portion of the same framework lies with the test scripts. Cribbing aside, from the design perspective, this approach makes both the test scripts incredibly bound to the framework, this coupling is solid and scraps away the capacity to scale up. This is the second flaw, the capacity to scale up is not present at all. As a result of this, the framework is always unstable. It never instills upon the user the confidence of proper running and proper notification. This is our third flaw, reporting, which is broken down into multiple layers, files upon files are created which should not be the way. Apart from this, the other issue that exists is also that of graph/forecasting capacity. This is also static, strict and very hard to scale up. This should never be the case.

There are a few other issues as well, but those cannot be attributed to the “framework” as such, much more related to the system in which the test script is being run as well as the issue with policies implemented in the system.

Proposal for a better testing framework

In order to scale up, remove any and every issue that causes the aforementioned issues, it is better to achieve 100% decoupling of the framework and the test scripts using this framework. REST API(s) might be the best way forward in this regard. The benefits are listed below.

• No SSH issue.
• Easy setup of framework in running condition in system under testing(hence forth will be called SUT)
• Easy workspace setup
• Development decoupling
• Easy reporting of issues
• Debugging capacity of test scripts as well as framework features.
• Easy identification and notification of issues.
• Pluggability and customisation of modules.
• Easy deployment.

The main thing is, the proposed framework will be re-using almost 70% of the code already present. Majority of work involves refactoring, re-arranging and wrapping up the framework features into related modules.