QA Challenge Accepted 4.0 conference was held on April 21, 2018, in Sofia at the International Expo Center. I witnessed overall good organization and interesting lectures. All slides were in English, while most presentations were in Bulgarian and some in English. This could be a little bit confusing if your English is not very good. Here are the highlights from the presentations.

Misconceptions About Software Testing

The first speaker was Claudiu Draghia from Romania. He presented in English.

A misconception is an incorrect opinion based on faulty understanding. We are all biased by our environment and team, and we have different perspectives. Here are some misconceptions, followed by reasons why they are incorrect:

1. Everyone can test, so testers should receive less money than developers. Can everybody drive? Everybody can try, but not everybody can be successful at it. There is a misconception that testing is easy. We should communicate better what testing is and what problems we are overcoming.
2. Testing can be done under all circumstances. Not everything can be tested. Software should be testable. Testability is an important feature of the software being tested.
3. Testing cannot be done without a specification. Exploratory testing busts this misconception.
4. Understanding requirements means reading them. Try to visualize, draw, and model them. Try to understand them piece by piece, one step at a time.
5. Testers break software. It is not true. QAs break the illusions about the software; it was already broken.
6. Testers are senior if they have five-plus years of experience. You could have done the same things all those years. You are senior if you can succeed in another team.
7. There is not enough time for testing. Reevaluate what has to be tested and reduce scope if your testing time is reduced.
8. Testers are bearers of bad news. You should be diplomatic. Try saying, “We are not done yet with this project.”
9. It is okay and good to test at the end of the sprint. Test one beat at a time. Start early. Otherwise, you will feel squeezed and not very effective.
10. Testing is what testers do. It is an activity performed by all team members in a variety of ways.
11. Testing is not a structured activity. It should be structured in your head.
12. Testing finds all bugs. This depends on goals, time, people, and resource constraints.
13. Production bugs are failures. You have missed bugs, and this is normal if you learn from your mistakes. You should not make the same mistake twice.

Deep Oracles: Multiplying the Value of Automated Tests

Emanuil Slavov talked about improving high-level automation tests. Usually, you use an oracle to determine whether a test has passed or failed. There could still be issues even if the test has passed. A deep oracle finds those issues and helps existing automation tests catch more problems.

Tests are more flaky higher in the test pyramid. Google has around 1.5% flaky tests; 1% is considered acceptable.

Some sources of bugs include:

1. Configuration issues, such as load-balancer configuration, connection-pool capacity, or external URLs missing from a CDN.
2. Application issues, such as thread-unsafe code, lack of retries in a distributed system, or database connections not being closed after use.

Use test data that is random but looks real. Different names, countries, and credit cards exercise different code. Pay attention to special characters in data, such as quotes, Twitter mentions, and URLs.

Use service virtualization. Testing with real third-party services such as PayPal or Amazon may not be possible or may be expensive. Tests should be able to generate all the data they need.

An attack proxy is an HTTP or web service between the test and the application. It can help find SQL injection, data disclosure, and other security issues. Emanuil showed how numbers in POST requests, API-token referrers, and JSON fields could be attacked. You should carefully assess whether to use your own tool or an existing security-testing tool; the tool needs to know a lot about your application to be successful. One example was an XSS bug in an email field that had no character limit.

The speaker mentioned Acunetix, Netsparker, ZAP, and Burp proxy with headless execution. Burp is a command-line Java tool that they use with a slight modification. Emanuil’s blog is emanuilslavov.com.

The next techniques need a dedicated test environment, such as servers or containers. Usually, a test relies on assertions, but code execution may continue after the last step logs.

Track exceptions

Look for unexpected exceptions in logs. The speaker’s team had failed to catch a parse exception in their Elasticsearch logs. Bad data is similar to an exception: it could be missing, unrealistic, duplicated, badly formatted, unsynchronized, or conflicting.

Context matters. Zero is a valid value in SQL, but in PHP zero used in an if statement means false. Because of a bug like this, database arithmetic progression killed their production server three times.

According to the speaker’s experience, 19% of exceptions were caused by bad data.

The last described technique was monitoring application metrics. Record application statistics after each test run. With fast tests, you can tie performance bottlenecks to commits. Look for increases in application-log lines or exceptions after each commit. Other useful metrics include the total number of database queries. Parallel execution with 12–16 threads can help catch deadlocks in logs.

How to Make an Agile Team’s Life Easier with Automation at Every Level – Testing of Microservices

Stanislava Stoykova and Atanas Georgiev talked about inheriting a large KPMG project. Multiple teams needed to work on it while the technology changed rapidly. They addressed these challenges with microservices: small applications that work independently and use their own data.

They used Azure Service Fabric, ASP.NET Core, Angular, EF Core, and NServiceBus. The speakers showed a registration microservice single-page application.

In their team, everyone tests. QAs know which test scenarios are covered by unit tests, and the team has API tests for frontend and backend communication. They started with Postman and Newman, then found AspNetCore.TestHost. Its main advantage is that it does not require a real database; it uses an in-memory database. Their API tests run for seconds after each commit.

The microservices approach is expensive and is suitable for big projects.

Performing Performance Testing – Why, Who, How – Step by Step

The speaker was Nedko Hristov, whose blog is nedko.info.

Before discovering performance-testing tools, he performed performance testing manually with six colleagues connected to a local machine.

Performance is an ongoing process that starts during planning. It is a mistake to test only at the end of a project: a performance bottleneck found then could require deep architectural changes and code rewriting. Unfortunately, performance is often underrated because project estimates are short or there are no performance requirements.

Know which metrics you need, and measure before and after changes. Nedko showed demos using JMeter, Grafana, and InfluxDB. Performance-test results are written to the database during execution. Grafana dashboards can have alerts so that you do not break the whole infrastructure.

Automating Video and Audio in a Video Management System

The speaker was Marta Magiera from Copenhagen. Video automation is difficult because you must account for different hardware processors, video codecs, video and screen resolutions, hardware acceleration, and several other factors.

Audio automation is also challenging because of different codecs, frequencies (1000 Hz, 3000 Hz, and 6000 Hz), and modes. Performance is extremely important. The team progressed from no automation to automation.

They use Ranorex for UI tests, Jenkins for CI, and NUnit. The team depended on hardware because hardware acceleration cannot run on virtual machines. A challenge was having a stable FPS driver, since not all cameras support all codecs. They use a fake driver and created test videos because two different videos cannot be compared; performance depends on the amount of movement in a video.

The team runs its tests nightly; execution takes 12 hours. It has 200 different tests that verify frequency-sample integrity. For audio tests, it uses VB-Audio Virtual Cable, Audacity, and Spek to measure sound quality. NAudio is an open-source .NET library that helps with automation.

Visual Test Automation in Adidas – Practical Usage

The speakers were Nikolay Stanoev and Georgi Yordanov. Their challenge was testing 45 custom sites in 30 languages. Visual testing was even more important to their customer than functional testing. They release daily, and German translations often break the layout because of longer words. Their websites have 50 payment providers integrated. In broad terms, visual testing compares two images.

They reduced manual visual testing by 30% and had zero critical or major production bugs after starting to use Applitools. The available comparison methods include:

• Exact pixel-perfect comparison
• Strict pixel-perfect comparison with color tolerance
• Content comparison
• Layout comparison

Tips

• Use layout comparison for a full page, or combine strict and layout comparison; use ignored regions for changing content.
• Use user journeys for visual testing.
• Reuse or convert functional tests.
• Limit test execution to 10–15 minutes.

Bugs found

In the first two months, the tool found 53 issues, including missing or new content, UI changes that broke the layout after library updates, and translation issues.

Benefits

• Maintenance cost is currently 0–1 minute per test.
• Fast results with no additional code increase confidence in releases.
• Maintenance is easy and the learning curve is short; a beginner can start using the tool in 1–2 hours.
• Some result checks can be delegated to designers or other colleagues, reducing unnecessary communication.

They take screenshots on real mobile devices. Applitools is a paid visual-testing solution. This topic was similar to the one presented by Nikolay Stanoev at the ISTA 2017 conference.

Cypress vs. Selenium

The speaker was Lyudmil Latinov, who made a comparison of the tools.

Selenium

• Supports bindings for many programming languages.
• Driver-browser communication resembles client-server architecture and is done through JSON.
• Selenium is a library and requires additional setup.
• Used mainly by QAs.
• Supports all browsers.
• Tests are relatively slow.
• Waiting for an element is not very stable.
• Supports parallel and remote execution.
• Has a great community.

Cypress

• Cypress runs directly in the browser and has direct application access.
• Developed for QAs and developers.
• Supports travelling back in time to a snapshot before every step.
• Is a complete framework containing Mocha, Chai, and Sinon.
• Can be used for UI and integration tests.
• Has only JavaScript bindings.
• At the time of the presentation, supported only Chrome; Firefox was in development and Edge was on the roadmap.
• Runs tests quickly, with almost no delay, and does not need waits.
• Does not support parallel or remote execution.
• Can record video and has excellent documentation, though its community was less developed.
• Cannot switch tabs.
• Can control browser network traffic and bypass security limitations such as CORS.

Both Selenium and Cypress are open source, and both can load extensions and manipulate cookies. Although Cypress was not as mature as Selenium, the speaker preferred writing tests with Cypress.

Automation vs. Intelligence – “Come with Me, If You Want to Live”

The speaker was Viktor Slavchev.

Why are testers so special that they cannot be replaced? Testers cannot be replaced by unit tests, methodologies, automation, or artificial intelligence. Speculation that humans will be replaced by AI is based on movies. Testing is not just a set of simple, executable, repeatable steps; it has a social nature. Quality and risks are social concepts, just like money.

AI is good at facts and rules, and partially at coaching. It is not good at creative tasks.

Team Motivation 3.0: Burying the Carrot and Stick

The speaker was Aneta Petkova. Motivation has a history.

Motivation 1.0 was the motivation to survive physically. Motivation 2.0 is the motivation to thrive in society: if you receive a reward, you do more of what you did; if you are punished, you stop doing it.

Some things defy this model, such as open-source software (Selenium, Linux, and JMeter), which is successful and used by millions every day even though its developers and maintainers receive no money.

The speaker described the candle experiment, which had reward and no-reward groups. The task was to use objects in a non-conventional way. The reward group needed longer to solve it because part of their brain was occupied with thinking about the reward.

Another example was a kindergarten that introduced a fine for late pickups. Parents started arriving later than before the fine was introduced, because they thought they had the right to be late after paying for it.

Money is not always the answer. It is like a drug: you need to increase the dose regularly to maintain its effect.

Motivation 3.0 is driven by enjoyment or accomplishment. This intrinsic motivation is connected with autonomy, mastery, and purpose. Being a team starts with building trust. Be on time and get the work done. Communicate what you want to accomplish. People are equal but not the same: recognize strengths and weaknesses, give people choices, and do not project your opinion onto others.

You manage things; you lead people.

Lightning talks

Anton Angelov mentioned the Meissa Runner tool. He was going to present it as a Selenium conference speaker in India that year.

Vasil Tabakov’s message was to spread quality into society.

Claudiu Draghia talked about learning from mistakes. We all make mistakes as humans and testers, and we are afraid of making them. This was not the case when we were younger. Learn from mistakes and do not repeat them. Nobody is perfect; prepare to make more mistakes.

Summary

This article summarizes presentations from the QA Challenge Accepted 4.0 conference. To learn more, visit the conference YouTube channel. Videos from previous years are available, and the organizers were expected to publish videos from this year soon.