28,79 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Fachliteratur
- Sprache: Englisch
From simple websites to complex applications, delivering quality is crucial for achieving customer satisfaction. How to Test a Time Machine provides step-by-step explanations of essential concepts and practical examples to show you how you can leverage your company's test architecture from different points in the development life cycle.
You'll begin by determining the most effective system for measuring and improving the delivery of quality applications for your company, and then learn about the test pyramid as you explore it in an innovative way. You'll also cover other testing topics, including cloud, AI, and VR for testing.
Complete with techniques, patterns, tools, and exercises, this book will help you enhance your understanding of the testing process. Regardless of your current role within development, you can use this book as a guide to learn all about test architecture and automation and become an expert and advocate for quality assurance.
By the end of this book, you'll be able to deliver high-quality applications by implementing the best practices and testing methodologies included in the book.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 544
Veröffentlichungsjahr: 2023
Ähnliche
How to Test a Time Machine
A practical guide to test architecture and automation
Noemí Ferrera
BIRMINGHAM—MUMBAI
How to Test a Time Machine
Copyright © 2023 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Group Product Manager: Rohit Rajkumar
Publishing Product Manager: Bhavya Rao
Senior Content Development Editor: Adrija Mitra
Technical Editor: Saurabh Kadave
Copy Editor: Safis Editing
Project Coordinator: Sonam Pandey
Proofreader: Safis Editing
Indexer: Manju Arasan
Production Designer: Ponraj Dhandapani
Marketing Coordinator: Rayyan Khan
First published: March 2023
Production reference: 2090823
Published by Packt Publishing Ltd.
Grosvenor House
11 St Paul’s Square
Birmingham
B3 1RB
978-1-80181-702-8
www.packtpub.com
For Hackerin, 2013-2022
For he was on my lap for the most part of this book.
May his loving soul rest in cat's heaven.
– Noemí Ferrera
Foreword
Noemí Ferrera’sHow to Test a Time Machine is a comprehensive guide to testing in the software development industry. This book is an excellent resource for anyone interested in software testing, whether you’re an experienced professional or a beginner. (I mean, the book title alone should make you want to read it!)
Noemí's approach to testing is unique and refreshing, focusing on topics often overlooked in traditional testing literature. In this book, she covers topics such as continuous testing, cloud testing, AI for testing, mathematics and algorithms in testing, XR testing, and testing patterns, among others.
Testing is essential to software development, and ensuring that software is reliable, secure, and performs well is critical. As the software industry grows, so does the need for high-quality testing. How to Test a Time Machine provides a unique, comprehensive, and much-needed guide to testing modern software.
The book is divided into several chapters, each covering different testing-related topics. The first few chapters cover the basics of testing and the test pyramid. The middle chapters delve into more specific topics, such as UI automation patterns, continuous testing, and AI for testing. The latter chapters cover more challenging topics, including testing hard-to-test applications and taking your testing to the next level.
I’ve interviewed Noemí many times on my TestGuild podcast, and she’s spoken at my online Automation Guild testing events, so I know she knows her stuff, and this book does not disappoint.
Noemí's writing style is engaging, making it easy to follow and understand complex topics. Her use of examples and practical advice makes this book a valuable resource for anyone interested in testing, whether they are beginners or experienced professionals.
In conclusion, Noemí Ferrera’s How to Test a Time Machine (and Other Hard-to-Test Applications) is an excellent resource for anyone interested in software testing. Her unique approach and comprehensive coverage of various testing topics make this book a must-read for anyone looking to improve their testing skills and knowledge.
Joe Colantonio,
Founder @TestGuild
Contributors
About the author
Noemí Ferrera is a self-taught programmer and wrote her first comprehensive program at the age of nine. When she grew up, she proceeded to achieve a degree in computer science specializing in hardware in Spain, a bachelor’s degree in software engineering in Ireland, and a master’s degree in computer science in Spain.
She is an international speaker and participates in testing community conferences (such as Selenium, Appium, and Automation guilds) and engages with the community through Slack channels, Twitter, and her blog.
In the 2020 Test Guilds publication, she was named as one of the top 28 test engineers to follow, as well as one of the top 33 test automation leaders to follow in the 2019 Tech Beacon and as one of the 100 women in tech to follow and learn from by agiletestindays.com in 2023.
There are so many people that influenced me in one way or another so that I could write this book, but I can't fit them all in here, so I need to summarize.
I would like to start with all my teachers that helped me enjoy writing, computer science, and English. However, the person that influenced me the most to learn and love this language has been my sister, Rosa. It's because of her influence that I decided I wanted to study abroad, which led me to work abroad, and to everything else that came after.
It feels like it was only yesterday that I joined the quality team at IBM, getting my first opportunity in the quality area. It would be too much for me to mention everyone I worked with, but know that you are also in my thoughts while I write these lines.
I also want to thank everyone in the testing community, especially those on the Selenium committee, who trusted and believed in me, even before I made any code commitment. Thank you, Maaret Pyhäjärvi, Lisa Crispin, Marit Van Dijik, Raluca Morariu, Sneha Viswalingam,Tristan Lombard, Diego Molina, Andrew Krug, Bill McGuee, Corina Pip, Manoj Kumar, Simon Stewart, and Marcus Merrell, for being inspiring, trusting, and welcoming. (And I'm sorry if I skipped you, there are so many people to thank here; you are still important!)
I really don't think I would ever have completed this book it if weren't for the Packt team. I would never have been happy enough with it. So a big thanks to eveyone in at Packt, especially Adrija and Bhavya, who cheered me up and kept me accountable.
I also wanted to thank Philip, the reviewer, who really took the book seriously and called out important problems with it, and Joe, who wrote the perfect foreword and made me tear up a little when reading it (who am I kidding? I was plain crying!).
I want to thank my friends and family for having the patience to deal with me while writing this book and when I told them I could not go out because I was behind on it.
Special thanks to my mom – she has always been there for me and her support and patience has been paramount.
About the reviewer
Philip Daye is a seasoned software quality professional with over 25 years of experience in the field. Currently the QA team lead at Insider Intelligence, he has a diverse background as a tester, manager, architect, and leader, and has worked with companies of all sizes to ensure the delivery of high-quality software. Philip is deeply committed to staying current with advances in the field and actively shares his knowledge and experience with others through speaking engagements at conferences and meetups, as well as by founding internal communities of practice.
I’d like to thank my wife, Shirley, and sons, Simon and Zachary, who are always my biggest supporters. To Tariq King, who has been a true friend and mentor, for his encouragement to continue to learn and share. To Dionny Santiago, Justin Phillips, Patrick Alt, Keith Briggs, and David Adamo, who challenged me to bring my best every day. Finally, I’d like to thank the author for the opportunity to be a reviewer of this wonderful book.
Table of Contents
Preface
Part 1: Getting Started – Understanding Where You Are and Where You Want to Go
1
Introduction – Finding YourQA Level
Technical requirements
Tips for absolute beginners
Figuring out which quality assurance level your company has
What development process is being followed?
What coding language to use?
What is the relationship between developers and testers?
What is the size of the company?
Questions about the development
How to get to the next QA level in a company:
Convincing upper management of the need to test
Convincing other team members
Getting a full framework up and running
Identifying priorities, improvement areas, and skills
Identifying improvement areas
Building the right team – testing roles and skills
Scaling
Automating the automation
Moving on
Summary
Further reading
2
The Secret Passages of the Test Pyramid – The Base of the Pyramid
Technical requirements
What is inside the base of the pyramid?
Unit testing
Other dimensions
The secret passages
How to cheat on unit testing to achieve coverage and why you should not
Statement testing
Edge testing
Loop testing
Branch testing
Error testing
Parameter testing
Data providers
Do not kill the mockingbird – how to level up your unit testing with mocking
How does mocking affect TDD?
The dangers of mocking
Secret passage – automation at the base of the pyramid
Summary
Further reading
3
The Secret Passages of the Test Pyramid – the Middle of the Pyramid
Technical requirements
What is inside the middle of the pyramid?
Integration testing
Service testing
API testing
Contract testing
System testing
Understanding API, contract, and integration testing
Testing APIs
Contract testing
Integration tests with APIs
The secret passages – shadowing testing
Shadow testing with an API
Shadow testing without an API
Static versus dynamic shadow testing
The dangers of shadow testing
What shadow testing looks like
The other side of the middle of the pyramid – performance testing
The performance... of what?
Load testing
Stress testing
What should a performance test look like?
Summary
Further reading
4
The Secret Passages of the Test Pyramid – the Top of the Pyramid
Technical requirements
What is inside the top of the pyramid?
End-to-end tests/user interface tests/frontend automation
Other dimensions
Secret passages
The secret passages – making your UI tests more efficient with API calls
Covering repetitive parts
Playing with test data
Alternatives to APIs
The secret passages – headless testing
The secret passages – remote execution
The search for other tests
Scaling your tests
When to write each T in TDD
Summary
Further reading
Part 2: Changing the Status – Tips for Better Quality
5
Testing Automation Patterns
Technical requirements
The POM for UI automation
Inheritance in the POM
A PFM – an antipattern?
Exploring a FOM
Handling localization
EPOM
RPOM
Putting it all together
Automating the automation – a practical example
Recognize automatable tasks
Write some code that does that task for you
Identifying when the code needs to be executed
Identify success measures
Dealing with dynamic object IDs
Items with dynamic locators
Angular/React dynamism
Summary
6
Continuous Testing – CI/CD and Other DevOps Concepts You Should Know
Technical requirements
What is continuous testing?
Continuous integration
Continuous delivery
Continuous deployment
Continuous testing
CI/CD and other DevOps concepts that you should know
Pipelines
Branches
The difference between a branch and a pipeline step
Secret variables
Feature toggles/flags
Load balancers
Rolling
Re-creating
Rollback
Metrics, alarms, and reports
Types of continuous tests
Unit tests/code review testing
Integration tests/system tests/contract tests/API tests
Build verification test
E2E/UI tests
Deployment testing
Performance monitoring
Tools for CI/CD
Programming languages
CI/CD tools
Packaging tools
Performance monitoring tools
Version control systems
Workflow applications
A basic CT example
1. Running tests programmatically
2. Logging
3. Creating a YAML file
4. Executing the commands
5. Getting notified
Summary
Further reading
7
Mathematics and Algorithms in Testing
Technical requirements
Algorithms, mathematics, and testing
Graph traversals
Understanding the role of data science in testing
Creating a dashboard
Reviewing some techniques for test case analysis
What should go in the BVTs?
The early approach to AI – it’s all thanks to mathematics
Identify repetitive tasks
Write code that would do these tasks for you
Identify when the code needs to be executed
Identify success measures
Summary
Further reading
Part 3: Going to the Next Level – New Technologies and Inspiring Stories
8
Artificial Intelligence is the New Intelligence
Technical requirements
Is AI taking my job? Why learn about AI?
Will automation take my job?
Why should I learn about AI?
Core concepts of AI
Types of learning
Data curation
AI for testing
Games
Data analysis
Test case creation and/or automation
Grouping tests
Automated visual testing
Smart DOM finder
Testing by voice
Smart test balancer
Optical character recognition (OCR)
Test case analysis with machine learning
Testing AI apps
Types of AI apps
Considerations about AI apps
Sending your code to the Gym
Other cool AI projects and tools
Other tools for AI
How to learn more about AI
Summary
Further reading
9
Having Your Head upin the Clouds
Technical requirements
What exactly is the cloud, and how can it help you?
Creating a benchmark to measure testing performance
Testing appropriately in the cloud
Parallelizing tests
Orchestrating tests
Dangers of the cloud
Considerations for creating your own device farm
Using test cloud providers securely
Testing applications in the cloud
Thinking out of the box about the cloud
When not to use the cloud...
Summary
Further reading
10
Traveling Across Realities
Technical requirements
Getting started with XR
How to get experience in XR
VR and testing
VR development
Testing in VR
AR and testing
Finding XR bugs and testing in XR
Base layer
Middle layer
Top layer
Other testing
Testing with XR
Quaternions and why they are important in XR
Tools for XR development
XR definitions
Sumerian [5]
Unity [8]
Unreal [9]
O3DE (Open 3D Engine) [10]
Other applications
Creating an XR application with Unity
The camera
Tools for XR testing
Unit testing in Unity
XR automation
Unium [21]
Airtest Poco [22]
Hardware automation
XR emulators
The metaverse
Summary
Further reading
11
How to Test a Time Machine (and Other Hard-to-Test Applications)
Technical requirements
How to test a time machine
By analogy
Bottom-up
Challenges in game testing
Automating the impossible
Automating for a higher purpose
Record and playback tools
Last-minute tips and best practices for your automation
Architecting a system
Summary
Further reading
12
Taking Your Testing to the Next Level
Technical requirements
Challenging your career
Finding hot topics
Finding your style – horizontal versus vertical knowledge
Finding time to work on that hot topic
Quality growth
Roles for testing
Helping people in your team grow
Transitions and career opportunities
Impostor syndrome
Closing thoughts
Finding your why
Ethics in testing
The future of quality
The future of you
Summary
Further reading
Appendix – Self-Assessment
Index
Other Books You May Enjoy
Preface
From simple websites to complex applications, delivering quality is crucial for achieving customer satisfaction. How to Test a Time Machine provides step-by-step explanations of essential concepts and practical examples to show you how you can leverage your company's test architecture from different standpoints in the development lifecycle.
You'll begin by determining the most effective system for measuring and improving the delivery of quality applications for your company and then learn about the test pyramid as you explore it in an innovative way. You'll also cover other testing topics, including cloud, AI, and VR for testing.
Complete with techniques, patterns, tools, and exercises, this book will help you enhance your understanding of the testing process. Regardless of your current role within development, you can use this book as a guide to learn all about test architecture and automation and become an expert and advocate for quality assurance.
By the end of this book, you'll be able to deliver high quality applications by implementing the best practices and testing methodologies included in the book.
Who this book is for
This practical book is for test owners such as developers, managers, manual QAs, SDETS, team leads, and system engineers that wish to get started on or improve their current QA systems. Test owners looking for inspiration and out-of-the-box solutions for challenging issues will find this book beneficial.
What this book covers
Chapter 1, Introduction – Finding Your QA Level, introduces the test architecture, analyzing how different projects can be built and how that can affect the test architecture. It also discusses ways of achieving and improving the test architecture, depending on the stage that a project is at, and identifies the different roles that can participate in quality, their responsibilities, and skills.
Chapter 2, The Secret Passages of the Test Pyramid – the Base of the Pyramid, reviews what type of tools, techniques, and testing are done in the base of the test pyramid, including the differences between coverage and good unit testing and mocking. It also shows a way of automating the base of the pyramid, making these tests easier and more fun to write.
Chapter 3, The Secret Passages of the Test Pyramid – The Middle of the Pyramid, reviews the faces of the middle part of the test pyramid, defining the differences between the backend tests and how they relate to eachother, along with exploring sending APIs, schema validation, sending messages, and shadow and performance testing.
Chapter 4, The Secret Passages of the Test Pyramid – The Top of the Pyramid, covers some of the main tests systems should have, going from the top of the pyramid to the bottom. It suggests some extra tips and projects you can use to improve your quality at the top of the test pyramid.
Chapter 5, Testing Automation Patterns, examines several models for UI automation test code writing to create better-designed frameworks. It visits different models, such as remote topologies, and deals with files with objects and screenshots, and reviews some ways of automating the repetitive parts of test code.
Chapter 6, Continuous Testing – CI/CD and Other DevOps Concepts You Should Know, introduces continuous testing, CI/CD, and other concepts related to DevOps. It discusses different types of continuous testing and tools for CI/CD, and provides a basic continuous testing example.
Chapter 7, Mathematics and Algorithms in Testing, highlights the importance of mathematics in testing, including algorithms that can be specifically useful. It reviews the importance of data science in testing and includes techniques for test case analysis, with an early approach to artificial intelligence (AI).
Chapter 8, Artificial Intelligence is the New Intelligence, discusses the role of AI in testing, including why you should learn it, core concepts, AI for testing, and testing AI applications, with some specific projects and examples to get you started and excited about AI for testing.
Chapter 9, Having Your Head up in the Clouds, explores the cloud, including how it can be useful for testing, how to create tools to measure testing performance, how to test appropriately in the cloud, and its dangers.
Chapter 10, Traveling Across Realities, reviews the different concepts and related applications of XR, including VR and AR, and what particularities they have in terms of quality verification. It includes some tools for XR development and testing in XR and briefly covers the concept of the metaverse.
Chapter 11, How to Test a Time Machine (and Other Hard-to-Test Applications), reviews tips and tricks for testing challenging applications, covering games, difficult applications, non-testing automation, and even what to do with impossible-to-test applications. It starts by discussing how to test a time machine (which gives this book its name) and how this could be extrapolated to any other app and concludes by talking about test architecture.
Chapter 12, Taking Your Testing to the Next Level, offers some tips on how to further your career, find topics that inspire you to reach the next level, and align them to your working and learning styles so that you can build your career plan and, finally, find time to work on it. It includes ways of helping others in your team grow and reasons for doing what you do and caring about quality, alongside the ethics related to testing and how these topics will apply in the future.
To get the most out of this book
While this book is meant to get everyone started in all the discussed topics, having some knowledge of programming is preferable as it will help you get the best out of the practical examples.
On the other hand, if you have computing knowledge, but not so much testing knowledge, you may find some of the examples too easy for you. Feel free to take them to the next level and try out the more challenging aspects of what is discussed here.
Software, programming languages, and libraries covered in the book:
Java, junit-jupiter:5.7.2, Python, unittest, TypeScript, JavaScript, C#, JSON, CSV, RabbitMQ, Cypress, Selenium 4, AirtestProject/Poco, YAML, 3D.js, pandas, Unium, Unity, NumPy, scikit-learn, sklearn, gym, and TestNG.
Additional instructions to set up each of these are provided within each chapter. Make sure you install the mentioned programs and libraries for each chapter.
If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.
Disclaimer
The author is not writing in representation of her company and no content in the book is related to any of her work within her company.
Download the example code files
You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/How-to-Test-a-Time-Machine. If there’s an update to the code, it will be updated in the GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/gFY0Z.
Conventions used
There are a number of text conventions used throughout this book.
Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Note that the following files can be located within the Benchmark folder in our GitHub’s repo.”
A block of code is set as follows:
package chapter2; public class Calculator { public int add(int number1, int number2) { return number1 + number2; } }Any command-line input or output is written as follows:
java -cp (path to the lib folder where testng is)\lib\*;(path to the bin folder)\bin org.testng.TestNG testngByGroup.xmlBold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “We need a system to keep all the code versions together. The place where code is kept is called a code repository.”
Tips or Important Notes
Appear like this.
Get in touch
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.
Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Share Your Thoughts
Once you’ve read How to Test a Time Machine, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.
Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.
Download a free PDF copy of this book
Thanks for purchasing this book!
Do you like to read on the go but are unable to carry your print books everywhere?
Is your eBook purchase not compatible with the device of your choice?
Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.
Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.
The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily
Follow these simple steps to get the benefits:
Scan the QR code or visit the link belowhttps://packt.link/free-ebook/9781801817028
Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directlyPart 1 Getting Started – Understanding Where You Are and Where You Want to Go
Each application is different, each company is structured in a different way and with different resources and budgets, and each one has a different maturity state. This does not mean they should aspire to go to the next state; maybe they are right where they should be. There is no one-size-fits-all solution. Think about quality architecture as a ladder where you want to focus on reaching the next step to the one you are on. Go one step at a time. This section is going to be the first few steps of that escalator, beginning with finding out where you stand and then reviewing all topics related to the test pyramid, focusing on the obscure things that other books might not tell you about.
This part comprises the following chapters:
Chapter 1, Introduction – Finding Your QA LevelChapter 2, The Secret Passages of the Test Pyramid – The Base of the PyramidChapter 3, The Secret Passages of the Test Pyramid – The Middle of the PyramidChapter 4, The Secret Passages of the Test Pyramid – The Top of the Pyramid1
Introduction – Finding Your QA Level
Each application is different, and each company is structured differently, with different resources and budgets. There is no ‘one size fits all’ solution with respect to quality. Instead, we can think about test architecture as an escalator on which each time you want to focus on reaching the next step to the one you are on now. However, to understand how to get to the next level, you first need to understand the level you are currently on.
In this chapter we are going to cover the following main topics:
Figuring out which quality assurance level your company hasHow to get to the next quality assurance levelIdentifying priorities, improvement areas, and skills needed to reach the next levelTechnical requirements
None! Take this chapter as your guiding star in testing and in this book. I hope that as you and your company progress in quality, the chapter will have a different meaning each time you read it.
Tips for absolute beginners
Whether you are a beginner at testing or even if you have an intermediate or advanced level of knowledge in testing, there are many ways in which you can find inspiration and knowledge: reading books and articles, attending talks, meetups, and conferences, attending courses, doing projects, participating in hackathons, working on projects, or playing around.
The test community is very friendly, and there are always people willing to help others.
The tools are already around those who are really committed to learning; the greatest difficulty is picking which one to use at any given time and being honest with yourself about how much time you are committed to spending on that.
This book might include concepts that are obvious to me, but you might be unaware of them. If that is the case, feel free to use the help of the references or search for those concepts before continuing to read.
I hope this book helps you find your inspiration and interesting projects to try out. Keep in mind Mark Twain’s famous words, “The secret of getting ahead is getting started.”
Figuring out which quality assurance level your company has
Getting started with a company’s testing requires making a series of decisions that could be hard to change later on. To be able to determine the best system for measuring and delivering quality for a company, you can start by asking some clarifying questions.
I also suggest you ask them whether you are about to join a new company or team to set the basis for your job and understand what to expect of the position and whether you will be able to grow professionally and help the company given its circumstances.
What development process is being followed?
Understanding the development process will help you understand how to test it better. Although companies are tending to move towards agile (for the reasons I explain next), there might also be companies that are perfectly content to be working with the waterfall system.
Waterfall
This sort of process is less common nowadays, but it is still in practice, for example, when systems are delivered in conjunction with hardware, or there are no expectations over new versions.
Testing a waterfall system usually implies managing a lengthy list of tests that are added up per feature and establishing planning for executing all those tests before launching the system to the user. We want to test everything that we can think of, and that can cover something that might already be covered with prior tests.
In these systems, it is especially expensive to find a bug in production as it might give a bad name to the product, could end up being dangerous for the end user, and might require building a new batch containing the product fix and having to re-deliver the product for free. You can see some examples of this in the Further reading section at the end of this chapter.
The tests are usually handled by one or more teams of experts, and they could be separated from the developers that built the deliverable (except for unit testing).
Although this is not so strict in a waterfall approach, starting testing early is always recommended. This is a lesson that we have learned from agile, as the earlier bugs are found, the easier they are to debug.
Agile
Although delivering bugs to the customers is something that could damage a company’s reputation and should be avoided at all costs, companies working in agile can afford to have minor bugs, provided they can be fixed quickly and seamlessly. It is important that the user has a platform to report bugs and issues if some of them reach the users.
Not delivering quickly enough could give advantages to competitors, lose clients’ interest, or even make the product irrelevant to the market. Therefore, the most important thing is to be able to balance the quality of the product with the quality of delivery time. This could mean having fewer but smarter tests. Testing something that was tested before will slow down each release. Since we have more frequent releases this could add up time. That said, every time an issue reaches production, it is important to make sure there is a new test in place that will cover this issue next time, while thinking about other tests we could have missed.
API testing gains importance over end-to-end testing in this sort of system (more on this in Chapter 4, The Secret Passages of the Test Pyramid – The Top of the Pyramid). This does not mean that long end-to-end, exploratory, or regression tests should not be done, but we can make use of the API to cover some of them (as we will see in detail in Chapter 4). Visual testing is also gaining importance in these systems.
To achieve more testing, the focus turns to deployment, as we could use servers that are out of reach for the users to do more extensive testing before deploying to them.
The relationship between developers building the feature and developers that are building automation or other members of the test team should be tight in this sort of system to avoid test repetition.
Fake agile
Many companies have moved from a waterfall approach to a fast waterfall that looks a lot like ‘agile,’ but it is not really ‘agile.’ In the process of the movement, we can see a lot of testing being carried over and trying to fit it into a smaller period for sign-off before release. These companies can be identified by having an impatient scrum expert or a project manager asking whether the test case was done and signed off every 5 minutes after the deployment had finished.
The best solution for this is to work alongside dev-ops or developers in charge of the deployments to set up a server system that would allow for more intensive, safer testing and figures out a way of cleaning the list of existing tests. In other words, we want to turn this into an agile system.
What coding language to use?
Unit tests should generally live in the same project of the class they test, using the same programming language.
If there are no tests in place yet, the advice would be to implement them in the same programming language as the developers are using. The reason is that if you turn to combined engineering or shift left in the future, the developers can take ownership of the tests. Sometimes this is difficult because you find that they are using different languages for the frontend and backend.
If there are any tests in place, the best thing is to follow along with what is there unless there is a major reason to rewrite them.
If there are too many tests in place that are not well maintained, then there are two ways of approaching this; refactoring or writing from scratch. Depending on how badly or well-kept the current tests are and how reliable their executions are, it might be easier to do the former or the latter.Here is a graph to help you decide which test code programming language to use:
Figure 1.1: Identify the most suitable test code programming language for you
The way the code is shared and controlled also tells you a lot about the system required, as does the task management, bug tracking, and logging or dashboarding systems.
Finally, if technologies used for development are changing, it would be a good moment to change the testing approach, and it is also crucial to have stable tests to make sure these changes do not affect the users.
What is the relationship between developers and testers?
As discussed in the previous sections, if developers and testers are well aligned, test repetition decreases, but independence might be affected.
Do they sit together in the office?
Being physically near shows that the company cares for communication between development and testing. If the teams sit separately, this could start to form an us and them mentality that could be seen in the product afterward.
For companies that allow remote working, the question would be whether they are in different teams or under the same manager.
Do they perform common code reviews?
Having a common code review system could result in finding bugs earlier. However, there is a general concern that having too many people signing off on a code review might slow it down or that the test team might not understand the code. I suggest that at least the unit tests are reviewed by someone with experience in testing, as code coverage tools could be easily deceived.
This also can have other benefits as all team members can improve their coding and testing skills.
What is the size of the company?
Bigger and well-established companies can afford to have specific teams or people dedicated to tool creation, automation, and quality.
On the other hand, startups struggle most to budget dedicated experts and prefer to find people who can work in multiple areas and adapt to each situation. They might prefer to deliver more bugs in exchange for quicker proof of concepts and faster development. You could argue that developing bugs and having to spend time figuring out what is wrong and where, is not at all faster than developing with superior quality from the get-go, but some of these companies have few, if any, users and are working iteratively to find the right product rather than focusing on building a product right.
The right time for a startup or product to start integrating testing earlier is when the product is about to be launched to users or when the focus changes to building the product right. Unfortunately, by this time, it might be difficult for a tester to start working on automation, as they did not participate in the feature definition and development plan, which could result in testability problems. It will also be harder to write the unit tests then. Therefore, the advice is always to incorporate testing into the development, make it a habit, and get the right set of code reviews and structural automation in place. In this way, the time added will be unnoticeable. We will talk more about unit testing, testability, and the benefits of developing with a test mentality in the next chapter.
Who writes unit tests?
This question might sound redundant, as nowadays test-driven development (TDD) is quite common, and most people understand the importance of unit testing and finding issues as early as possible. Nonetheless, this is not always granted, and some companies prefer someone other than the person that built the code to write all code for testing, including unit testing if needed. If that is the case, the test framework should include unit testing, and these tests might reside in a different place than the feature code.
What are the key test roles?
Some companies might have quality assurance (QA) people, test automation roles, and tool creators, or it might all be done by the developers, with sometimes QA being done by the product owners.
Not all companies assign the name software developer engineer in test (SDET) or QA to the same role. Therefore, it is important that you understand what each role involves and what the expected responsibilities are to be able to help achieve better quality.
It is also important to understand the number of people that are working exclusively on quality versus development, to understand the amount of automation that will be needed.
What does the app or company do?
This might seem obvious, but sometimes it is taken for granted. Depending on what the app or company does, you can think of the right and highest priority type of testing for it. For example, if the app deals with payments, localization and currency, testing is crucial.
How many users do you have or plan to have?
The current versus potential growth is as important for testing as it is for development. Understanding the amount of testing that might be done and planning the best ways of debugging issues found during testing is not always done the same way for smaller projects as for bigger ones. Similarly, we should plan the right testing environments, which might need dedicated servers or other resources. There could be specific problems, such as multi-engagement across users, and performance testing could be of higher importance.
Questions about the development
Having clarity about the way development is done in the company is highly important to create appropriate tests. We looked at development techniques before, but within these techniques, there could be differences in how the features are created. The following questions could also help clarify the quality process.
How frequent are the project iterations?
The more frequent the iterations, the less time to perform testing. If iterations are very frequent and we find the need to do exhaustive testing, we need to be creative about it. For example, having some production servers disconnected for the users with the new code and applying testing there while other servers are being built before distributing them to the users.
How long does it take to verify builds, and what is the goal?
Continuous integration/continuous delivery (CI/CD) is the most automated way of doing this, but the tests could have different degrees of extensiveness depending on the goal.
Is the priority aligned within the company? Per build or per feature?
Is there a clear list of features per build? Is there a straightforward way of tracking which old builds could be affected by new ones? If not, regression tests will need to be more extensive (it is good practice to regression test everything but there is not always enough time for it).
As you can see, successfully defining the right quality process requires a lot of different inputs. There could be more parameters that could help gain a better understanding of the particular needs of a company or project for quality, but by now, you should have a better idea of what works best for each case. In the next section, we will see three examples of common traits that I have seen in the past and analyze how they could be improved. Additionally, you can see a self-assessment in the Appendix.
How to get to the next QA level in a company:
In my experience, when someone asks for help getting started with testing, there are three statuses the company could be at, and therefore three things that might be needed:
Convincing upper management of the need to testConvincing the other team membersGetting a full framework up and runningConvincing upper management of the need to test
As much as you are convinced that a particular set of tests are important if management believes otherwise, all testing efforts would land outside of expectations for the feature delivery, impeding the proper planning and delivery of tests. The first step before building a test framework or incorporating a new type of test is communication.
Managers’ and product owners’ main concerns usually center on time and budget. Addressing them is not always easy, and it highly depends on each case. The most important question is “why do we test?” or “why do we need this test in particular?” Make sure you can explain it and that you can add data to support the case.
Convincing other team members
Similarly to convincing upper management, other team members might understand the need to test but not agree on where and how to do so.
If there is no test team in place, the best thing would be for the developers to start getting into testing by incorporating unit testing and TDD.
It is always nice to have someone with test experience to orchestrate tests, educate the team about testing, and start creating a test system for end-to-end and other sorts of testing or even perform some degree of manual checks. The same people or different ones could do this.
If the developers have never created any tests and/or are accustomed to other people verifying everything for them, it might be difficult to educate them on the use of unit testing and testability. But, as before, having enough data could help in the path of reasoning, even if that means you write some unit tests to prove their need.
If the issue is that they do not consider testing challenging enough, try to make them passionate about it. Maybe find some challenging test problems; there are some of them in this book.
Lastly, it might be that unit testing and mocking are taking as much time or longer than the time spent on developing the feature. If this is the case, consider automating some of it. (Yes! This is possible, and we will dive deeper into this in the next chapter.)
Getting a full framework up and running
This is required when the company has some unit testing and/or manual testing in place but still does not have a test framework for a particular test type. Alternatively, they might have one but feel insecure about the current tests and want to make sure they are not missing anything to improve their system.
I will be explaining how to set some of them up throughout this book, but there is plenty of information online, comparisons, trials, and opinions, so do not feel obliged to set up the same frameworks that are here; pick the ones that suit your system the best.
Identifying priorities, improvement areas, and skills
You should now have a better idea about the possibilities of testing around companies and applications. In this section, we are going to review some methods for improving each of the situations.
Identifying improvement areas
The test mentality has a lot to do with asking questions and being curious. We started the chapter asking some questions, and here are some more to help you discover areas in which you could improve around quality:
Is there any repetition that could be reduced? If so, what? How could you automate those processes?
How many tests do you have of each type? We will be talking about the test pyramid in the following chapters; what does yours look like? Are there any other types of tests that could be beneficial for you? Are there any tests that are at the wrong pyramid level (or in more than one)?
How long does it take to sign off for each deployment? The ideal time should be under 15 minutes (which does not mean you cannot test further in other environments prior to or after deployment).
How much do you rely on your current tests? Are we testing what we should be testing? For example, I’ve frequently seen tests related to downloading documents from the browser. Unless your app is a browser, you should trust the download works properly. Is there another way of testing that document without going through the pain of downloading it? Are there other tests in which you are testing functionality that you do not need to test?
How much do you trust in your tests? If tests are not reliable or out of date, are they adding any value? Keeping the system clean is highly recommended. If you have a version control system, if a feature ever comes back, you should be able to retrieve the old test code from the version history.
Are you using any tools to track the issues and understand where we need or do not need tests?
Do you have the right tests for the right time? It is important to understand what tests to run and when to run each throughout development. We will discuss this further in the test pyramid chapters. We should also make sure we understand why we are testing or why we need a particular test. Avoid failing to do something just because other companies are doing it or imposing a set of tests that are not needed yet.
Lastly, if it is still hard to discern when something is needed, I highly recommend talking to a professional; some people would consider a short consultation for more tailored advice.
Building the right team – testing roles and skills
Let us just take a bit of time defining testing roles as I have found that companies do not seem to agree about their definitions, and it is important to understand what people I am referring to throughout the book. I will also add some tips to help each of the roles grow in Chapter 12, Taking Your Testing to the Next Level.
Having a test expert help figure out the maturity of the company and what is needed to improve the quality of the product is particularly important. Test managers and test architects should be in distinct positions. However, not all companies need both positions, and sometimes, the job can be done by the same person. In some cases, automation is performed by developers, other times by developers in test, and QA, they are even called “automators” (which I believe to be a made-up word).
Rather than thinking of the following as “job positions,” you could also consider them “roles” that can be performed by different professionals as needed.
Test manager
A test manager makes sure the tests are performed correctly (created by the test or dev team) and good practices are implemented. They need to understand how and what to look for in the quality area. The position requires deep knowledge of testing and people skills.
Test architect
The architect designs the frameworks and tools to be used for testing and can give practical development advice to people building test code. This position requires deep technical knowledge and experience in planning and coding tools from the start while having deep knowledge about testing. Sometimes this position is covered by an SDET.
Software development engineer
Software development engineers (SDEs) are also known as developers. They are the people in charge of building features and, depending on the company, in charge of more or less of the test code.
Manual testers
Some people refer to manual testers as QA testers. They are knowledgeable and passionate about applications, issues that could arise, and test methodologies. However, in some companies, QA testers also write some automation (generally for the user interface (UI)). Some companies invest in teaching automation, providing tooling to help them achieve more in less time, including the use of automated behavior-driven development (BDD) to turn the test definition into code, and visual testing in which UI screens are compared automatically with an expected image.
SDET
SDETs are a rare species. They have the dichotomy of being developers with a testing mentality/passion.
Being stuck in writing test code could be frustrating for most developers as it is a repetitive task, not always challenging. When a company uses SDET to define the role that I am here referring to as QA, some people find themselves in that position, expected just to write automation code, have an unpleasant experience, and move away from the title.
Instead, they should be empowered to identify processes to automate and tools they could write to keep improving their programming skills.
Many companies are starting to join a movement called “shift left” or “combined engineering” in which SDE and SDET are combined into the “software engineer” role and work on all tasks related to coding, even test coding.
DevOps
DevOps is a combination of developers and operations. A while back, tasks related to servers, deployments, and networks were done by a specialized team, sometimes referred to as “systems engineers.” Most of the time, this also included a degree of system testing and even security.
As more programming languages, tools, technologies, and techniques started to develop, roles started to get increasingly specialized. However, the issues discovered by this team were usually difficult for them to fix, as they were not part of the development and needed an understanding of the code and features that they were not familiar with.
Here was where DevOps was introduced, in other words, developers doing this job for their team rather than across the company. In some companies, the “ops” bit is taken for granted and eliminated from the word, which I will do throughout the book.
Other terms
Other terms related to testing are systems engineers or system verification testers (SVTs) (like “ops” but with more test knowledge and debugging capabilities), functional verification testers (FVTs) (a rather old term that involves end-to-end front and backend automation and testing), and integration testers.
For simplicity, they will all be referred to as SDETs in this book, even though there might be SDETs specialized in some areas as developers are.
Scaling
Horizontal scaling means that you add more machines to your testing. For example, using the cloud to test in different systems (more web browsers, mobile versus desktop, and different operating systems).
Vertical scaling means that you scale by adding more types of tests (security, accessibility, and performance). We will talk about the “test pyramid,” where each test falls on it, and what percentage of the time spent in testing should be done in each of them.
Identifying the type of testing you need and what systems should be covered should be part of an architecture design. We will talk about diverse types of testing and tools for horizontal scaling in Chapter 2, Chapter 3, and Chapter 4.
Automating the automation
Time is the highest-valued currency you have; it is something you can never get back. This is one of the reasons why I like automation, so I can save time (mine or other people’s). And by automation, I do not just mean test automation but any repetitive process. That is what I mean by “automating the automation.”
In most companies, the “task automation experts” are the SDETs. That said, if you are a developer or a QA tester, you could highly benefit from this practice.
I have identified some basic steps for automating anything:
Recognizing automatable tasks:The first step to automating something is to think about repetitive tasks. Then, you should consider how long it would take to automate that task and calculate how much time it would save if it were automated.
You can tell when a company is doing well on their test automation when you see this thought process translated into it, rather than automating as many things as possible as some proof of skills or performance. The same concept can be extrapolated to any other repetitive tasks, including, of course, automation.
Write some code that does that task for you:Once you have a clear picture of the steps involved in the repetitive task, you should also have an idea of how to automate those steps. The exact language, style, and patterns are up to you to define.
Identify when the code needs to be executed:Sometimes we want to execute the code after something else happens. For example, testing automatically after a feature has been developed is common. Or, we can have automation that depends on some trigger, such as a page being refreshed or an application launched. Other times, we want the execution to happen at a certain point in time, for example, every morning. That could be automated with a cron job (also known as scheduled tasks) or services.
Identify success measures:The next step is to identify our gain from this automation. What do we need to achieve with it? What is our best result metric? Sometimes, to verify that the automation has been executed and to check its success, we rely on logging. However, checking the logs could also be considered a manual task, and we will need to make sure we automate it if that is the case.
I suggest creating an alert if something has gone wrong, for example, a test case failing. We may also have a notification that everything has gone well, too, just to verify it has worked. An email with the details, a text message, or even a phone call could all be ways of automating the logs (more details on notifications in Chapter 11, How to Test a Time Machine (and Other Hard-to-Test Applications).
Where to start?
When there are a lot of things a company or team could improve or even automate, I have a formula for automation that we can apply to other things:
(Time spent in doing the task manually + monetary value (based on the cost of potential issues)) / (time that will take to build the automation).
I argue that we could also think of it this way:
(Time spent in doing something before a change + monetary value (how much the task is needed)) / (time that will take to implement the change).
For “time spent,” I mean total time, if there are three steps, we need to multiply it by the potential times those steps will be repeated. Of course, there might be other factors to consider as well, for example, the scalability of current versus future solutions and people impacted by this, but I found this to work for general cases and as a baseline.
Moving on
Imagine that all the feasible options in this chapter are checked and implemented, and one or more test frameworks are set up. The team is managing the new tests, which are carefully planned and executed as part of the CI/CD structure. The analytics are set up and create alerts for any problems or improvements. So, can you ever be done with testing?
I presume most people in quality will jump straight away to this answer: “you are never done with testing.” However, in the same way as you might “be done” with development, you need to “be done” with testing. It is normal if you are not 100% happy about your project, but you have done your best, and (usually due to budget) maintenance is all there is left, at least for that project (and until some cool tool or technology comes along).
What to do then? As with development, at this point, we can move on to the next project. Of course, someone should stay to fix potential issues and provide support, but you should not need such a big team or even the same skills for this. Not worse or better, simply different. Some people enjoy being at the beginning of the project, and others prefer maintaining them.
Development and testing go hand in hand; when there are fewer changes in development, there are fewer changes in test code.
Summary
In this chapter, we had an introductory overview of the test architecture and analyzed how different projects could be built and how that could affect the test architecture. Since every company is different, we need to take into account all the possible inputs to produce the right set of testing. We can conclude that as development is intertwined with testing, decisions should be made for both of them at the same time.
Then we discussed ways of achieving and improving the test architecture depending on where the project is at that point in time. We considered what can be automated and how, and what process automation follows.
We identified the different roles that can participate in quality, their responsibilities, and their skills.
In the next chapter, we will discuss what is called “the test pyramid” and we will analyze its base. The base of the test pyramid is usually handled by developers, so they may be the most interested in this chapter, although SDETs and QA testers interested in growing in their careers might find automation that they could do in this section.
In the two chapters after the next, we will talk about the middle and the top of “the test pyramid,” discussing in detail different test types and some tips and tricks around them.
Further reading
Two examples of critical defects can be found with details at https://cloudwars.co/ibm/bank-of-america-cloud-slash-billions-in-it-costs/andhttps://techtrends.com/2019/04/19/a-software-bug-caused-boeings-new-plane-to-crash-twice/.
2
The Secret Passages of the Test Pyramid – The Base of the Pyramid
Test strategies have been explored throughout the years, with the common goal of finding the right amount of testing. The questions of why we test, what we test, and more importantly when we stop testing have puzzled test experts for years, resulting in almost philosophical conversations.
Testing has been divided into many groups and categories, the most famous of them being
