In the previous article, we explained what performance tests are, why they are performed, what types there are, and which one to use in a particular situation. Now we can move on to the next stages of testing, where some important questions need to be answered. We will first focus on test preparation, then test performance, and finally we will try to draw conclusions.
Frequently asked questions about performance testing
What is the product developed for and how is it used by users?
Each product has its own characteristics and can serve different purposes. There are products that only display content for the user (e.g., a blog, an information page for potential customers), but there are also those that allow you to perform actions other than just viewing prepared data (e.g., creating an account, signing in, making purchases, generating a report). Do we consider just a website or a dedicated mobile app or maybe both versions at the same time? From the perspective of what users do, it makes sense to put yourself in the shoes of one of them and try to predict their behavior.
How is the product built and what technologies are employed?
It's worth digging into the details and verifying that the frontend, backend, and infrastructure are properly prepared. On top of that, you should check how individual components are connected to each other. Are the technologies used outdated? Are they using obsolete and unoptimized components that can overload the system?
What are the relationships between processes and functionalities?
By performing even basic operations as a user, you can see some correlations. If you are interested in some item and are thinking about buying it, you want to visit the product page. On the product page we see 10 images, plus 10 thumbnails are displayed. Thumbnails are created depending on whether you are using a computer or a smartphone and can adapt accordingly. Will 100 people visiting the same item be able to view all the images? Are the thumbnails re-generated each time the page is visited? Is there a process that controls the thumbnails being created or prepares them ahead of time so they are available faster?
Another example: if we place an order for this item, the system needs to generate and send 3 emails. Assuming there are currently X users placing an order, this requires 3X emails to be sent to all interested customers. Is it possible to send all the emails simultaneously? Is there a process that controls the sending of emails and can set them aside?
You may even want to map out these relationships to better understand the complexity of the processes.
Source: https://c4model.com/
Source: https://c4model.com/
What functionalities might cause a problem?
As a product user, you may face situations every day when some functionality is noticeably taking too long to respond. Generating an invoice or a financial report for a whole month can be an operation that needs more server resources than other operations. During performance testing, it is worth checking if there are user actions such as downloading a large amount of data, aggregating data and sorting the results, or reading data directly from disk.
Is there a simple way to access the environment being tested?
The test environment can usually be accessed through security mechanisms such as VPN or .htaccess. Make sure that these security features will not affect your testing.
Can the tests be run seamlessly from a local machine?
For simple tests and a small amount of data transferred, you shouldn't have a problem using your own machine. However, if you are developing more demanding tests, consider moving the tests to more powerful hardware or using a cloud service so that additional factors do not affect the test results.
Are there any assumptions about the expected results?
Assumptions can come from observations (statistics) or business requirements.
The first approach is to collect data to help serve as a guide for testing. When we start planning performance tests, it is helpful to simply have some data to compare. Ideally, performance tests should have been run before. You can then go back to the documentation from previous tests and verify the information contained therein. Naturally, you should keep in mind that the test assumptions and product specifications may have changed, so you should adapt the new tests to the new requirements.
If there is no such documentation and the tests are just being developed, there is a way to do this as well. You can get the required data from various popular analytics tools, for example, observe the average user traffic for the last month with analytics tools like Google Analytics, Hotjar, HubSpot. You can also see how much interest the store had during the last Black Friday or during the entire holiday season. In case we run into problems because a server went down during the sale, analyzing this event can also give us a lot of information about how our tests should look like.
Another approach to determining assumptions is to get them from the business. The requirements are precisely defined in the project documentation, for example, the API must respond in < 50 ms. This means that the results must meet the requirements regardless of the number of users, new functionality or changes in the system architecture.
What environment should you perform testing on?
The testing should preferably be done on an environment other than production environment, since we usually don't want to affect the performance of the product. A misconfigured test can lead to unexpected results. If you need or want to perform testing on a production environment, discuss this with the team. The environment must be very similar to the production environment for the results to be reliable. You should also ensure that your environment is configured, that your code is updated, and that you have a copy of your database from the production environment.
It is important that:
▶ the environment configuration is similar, although some of the processes may change;
▶ the code version is properly prepared (so that there are no accidental changes);
▶ the database has the same range of data as the production database, e.g. the number of registered users or available products should not differ;
▶ you anonymize all data when making a copy; if you don't, real customers may be inundated with emails, for example.
What do you need to keep in mind?
▶ When you start preparing, you need to think about the purpose of testing and why you want to do it. Remember, testing can be performed for an existing product or an entirely new product.
▶ In addition to this, it is important to know if the new product meets certain initial assumptions or if the existing product has not deteriorated its previous performance.
▶ You can't skip the analytics which is extremely important to both the team doing the testing and the people who make the eventual business decisions.
▶ Performance is verified in terms of, e.g., correctness of responses, optimization of database queries, system speed and stability. This job should be done by testers, developers or DevOps.
▶ The result is not just a numerical value, but more importantly the real impact of the product on the Customer Experience. The outcome achieved is important from a business perspective as it allows you to determine whether your product is ready to meet user expectations.
▶ Simulation of the natural functioning of the site over a long period of time or a sudden change in the standard load? Analyze several scenarios that will help you choose the right type of test.
▶ The simplest and most important point – remember that the user will be happy to use the new functionality, but if they find it difficult to work with, they will look for a better solution among your competitors.
What's a good thing to do after testing?
1. Document the tests performed.
2. Describe the assumptions and effects from the tests performed.
3. Describe the causes of the problems identified and the methods used to fix them.
Why? The description may be useful for your next performance tests as you will be able to protect yourself in advance against known threats.
4. Prepare a checklist that points out the key elements needed for proper test performance.
5. Develop disaster recovery plans in case the server goes down, for example.
Practicing an emergency scenario under controlled conditions will be a great test for your project. You will find out if you have the right tools, facilities, if the right procedures are in place and, most importantly, what is the response time from problem detection to application recovery. If a database goes down, what is the recovery time? Is restoring the database enough or are there other steps needed? What might be the consequences of not having updated data from an external source, and how long does it take to get it?
What is the value of the performance tests performed?
- for TESTER
Tested functionalities meet functional and non-functional requirements. They work correctly for both single and multiple uses at the same time.
- for DEVELOPER
The functionalities are developed in a correct way. They are also optimized in such a way as not to disturb the stability of the product.
- for DEVOPS/ADMIN
The product environment is stable and the security measures implemented ensure proper operation even with increased traffic.
- for BUSINESS
The product being developed is ready for customers to use.
- for END USER
The product meets expectations, provides a comfortable user experience, and increases confidence in the brand. This makes customers eager to use the product next time.
Performance testing is necessary before any software is launched, and the cost of testing is usually offset by greater customer satisfaction. By optimizing your system with performance testing, you ensure that your website is ready for new traffic and orders, and that it will always be accessible to users.
If for some reason performance testing is still not performed in your project, it is worthwhile for the team to review the performance testing manifesto. It contains one of the most important postulates – even the simplest testing is better than no testing at all. Simple tests can quickly answer basic questions, highlight problems, and be the reason why the team will want to develop these tests in the future.