Let’s try to talk about Quality Assurance!

Quality Assurance, Quality Engineering, Quality Control, Test Engineering, Test Analysis, etc. Do you know the difference between them? Are there differences between them? Do the companies position, value, or define them correctly? Are they all engineers?

This is a very controversial topic and there is too much to talk about regarding Quality Assurance because there are too many wrong definitions, positions are in place and people are not valued correctly most of the times. I will try to touch some of them with my limited and humble experience. Let me know if you think I missed a topic or should have involved more on it.

First, we need to talk about the basis. Let’s start with mentioning three aspects needed for software quality:

• Measuring quality
• Improving quality
• Maintaining quality

The IT industry has reached a maturity level where controlling both the quality of a software product and maintaining a certain quality level are taken into serious consideration. We are past the point of solely focusing on the features of a software product. Nowadays, the success of a software is also determined by its scalability, maintainability, efficiency, and performance. We can group these parameters under the “quality” umbrella because they define the quality of a software product.

After building a strong mindset for the quality of your processes, next question is how to apply these. This brings us to the second topic I would like to touch; automated and manual testing. I would like to start with asking an important question. Should we or can we automate everything?

Definitions

Let’s start with some basics. According to West Virginia University (Example Career: Software Quality Assurance Engineers and Testers, n.d.), these are some of the things Software Quality Assurance Engineers should be doing:

• Design test plans, scenarios, scripts, or procedures.
• Document software defects, using a bug tracking system, and report defects to software developers.
• Conduct software compatibility tests with programs, hardware, operating systems, or network environments.
• Provide feedback and recommendations to developers on software usability and functionality.
• Design or develop automated testing tools.
• Coordinate user or third party testing.

The list is long but the brief summary is the job consists of 3 main approaches:

1. Automating Quality Assurance tests where possible
2. Conducting manual tests to create testing plans and form the happy flow as well as edge case scenarios
3. Collaborating with developers and field staff on creating and maintaining a quality mindset

When should automated tests be considered?

According to Test Guild by Joe Colantonio (Colantonio, 2022), some of the tests that can/should be tested with automated scripts:

1. Tests that need to be run against every build/release of the application, such as smoke test, sanity test, and regression test.
2. Tests that can be used for performance testing, like stress and load tests
3. Tests that involve inputting large volumes of data.
4. Tests that need to run against multiple configurations — different OS & Browser combinations, for example.

When should manual testing be considered?

According to an article by the BrowserStack community (Pai, 2021), some of the scenarios where you should stick to manual testing:

• When flexibility is required: Finding out the happy flow and edge cases to be tested for a new feature requires flexibility in testing. Automated tests need time and effort to build up, therefore, they should be involved in the later phases.
• Short-term projects: For time-limited projects or temporary solutions, the cost of building automated tests and maintaining them can exceed the value they create.
• When end-user usability is being tested: When the subject is looking or feeling good enough to human user, no machine can decide better than human input.

Cost of software automation

In order to understand the costs of software automation, we need to focus on the points below.

• Tooling: Automation tests may require licensed software, as well as dedicated hardware (server, cloud, database etc.).
• Process: The process starts with building/using frameworks, libraries, and artifacts. Each time the requirements change for a software, the structure might need to be changed/updated as well. This might cause the work to be repeated multiple times.
• People: Considering we work with limited budgets, putting more focus on engineers to create, update, and maintain automated tests code can cause us lose domain expertise and flexibility when we are faced with agile product requirements.

It’s clear to see that the increasing trend of “let’s automate everything” is a lost cause because Test Automation can’t test the user experience, user behaviour, product usability, product design, etc.

Robotic tests lacks the cognitive abilities of human input and therefore scores lower when conscious decisions have to be made. We need Manual Testing and Quality Assurance Specialists’ eye for detail.

We need the best of each approach and we need to combine their powers.

Should the same person be doing both manual and automation or are they different roles?

We have settled that we need both automated and manual testing but should the same person be doing them both? Naturally, we can consider them as different career paths under the Quality Assurance job family. In an ideal setup, we might want experts on each path so we might keep people limited with one to create more focus and expertise. However, since companies mostly don’t have unlimited budget and/or human-force, they prefer a hybrid approach, which consists below aspects on different weights:

• Automating testing processes
• Coaching developers to test their own work and cover more with unit/integration tests, shifting left when possible
• Manually carrying new stories alongside the Software Development Life-cycle
• Participate in creating company-wide testing strategy and plan

It might seem unfair to expect all aspects of Quality Assurance from a single engineer, but it is a widely accepted approach. Therefore, it might be better for Quality Engineers to know all of them but apply them smartly based on the business case.

Salaries and Job Families

Let’s go back to the point we made at the beginning. Clearly, there is ambiguity and confusion with the definitions under Quality Assurance job family. This profession has so many different names, titles, positions and job definitions that it’s hard to keep track of them all. One thing is common in many companies that people in these role are generally undervalued and sometimes wrongly positioned, due to a lack of understanding regarding what they are actually supposed to accomplish.

Let’s get a different perspective and check some numbers from Glassdoor. I used Amsterdam as the location for the salaries below:

• €82.000: Senior Software Engineer
• €82.000: Senior Software Developer
• €80.995: Senior Software Development Engineer
• €75.516: Senior Mobile Developer
• €66.131: Senior Web Developer
• €91.711: Senior Quality Assurance Analyst
• €74.135: Senior Software QA Engineer
• €69.569: Senior QA Engineer
• €68.507: Senior Test Automation Engineer
• €63.287: Senior Software Tester
• €61.553: Senior Software Test Engineer
• €61.513: Senior Test Analyst

The reason I wanted to mention these numbers is that even though they do similar things, people can be positioned far from each other, based on different titles or definitions within companies.

I have put Software Engineering and Quality Engineering roles together not to compare their worth, but to show that Software Engineering roles are more or less structured similar where Quality Engineering ones are too far away from each other.

A side note, I personally believe Software Engineers and Quality Engineers both share equal parts to the success of a software product; therefore, their valuation should also be as close as possible.

Conclusion

Going back to our main question, can the quality of a software product be assured only with automated tests? The answer is no, because to cover all the aspects, we need also manual testing involved within the process.

A good Quality Assurance Plan and Strategy should have them both. Depending on the circumstances, one can be more useful than the other but we should not restrict ourselves and be open to both. Both are required in the perfect solution.

References

1. Example Career: Software Quality Assurance Engineers and Testers. (n.d.). West Virginia University. Retrieved September 20, 2022, from https://www.wvu.edu/academics/careers/software-quality-assurance-engineers-and-testers
2. Colantonio, J. (2022, August 17). 15 Reasons Why You Should (or shouldn’t) Automate a Test. Automation Testing Made Easy Tools Tips & Training. Retrieved September 20, 2022, from https://testguild.com/what-to-automate/
3. Pai, A. (2021, December 31). Manual Testing vs Automation Testing : Differences. BrowserStack. Retrieved September 20, 2022, from https://www.browserstack.com/guide/manual-vs-automated-testing-differences