I was pleased to attend ISTA conference 2017 in Sofia, the capital of Bulgaria. My previous article was about the presentations on the first ISTA day; this article covers the lectures I attended on the second conference day.

Are You Managing?

Steve Odart’s presentation was about management and the balance that should be achieved in every project. 94% of all bugs belong to management because it failed to prevent the problems. Most managers do not receive formal training; they must learn on the job, so many follow patterns they have seen before. There are three main questions every manager should be able to answer:

1. Do you know what you want?
2. Do you know if you are getting it? Measure progress so you know whether you are achieving your goals.
3. If not, what are you doing about it? If you are not on the right track, change something.

Find the balance between the triangle of time, resources, and project scope. There should also be balance between coding features, technical debt, and continuous improvement.

We are in a bad situation when debt is increasing. Debt is created by:

• Deferring defects
• Poor architecture or implementation
• Poor quality control and lack of automation

Your goal should be to get a little bit better every day. When you try to motivate your manager to change something, show the effect of the change in terms of money.

The value trap is thinking that the more features we deliver, the more money we receive. A feature is not only code; it is also documentation, support, and automation.

The future of management is that we will all become managers. There will be self-directed teams with a coach, not a manager.

The Ultimate Feedback Loop: How We Decreased Customer-Reported Defects by 80%

Emanuil Slavov shared his experience about the root causes behind incidents reported by customers over 2.5 years. He was able to collect statistics because the team linked defect fixes to customer tickets in Jira. He asked, “If a defect goes unnoticed by customers, is it a defect?” There could be different points of view, but if a customer reports a defect, the team should investigate it.

The team did not have many algorithms in its code, but had 60–70 services. The most expensive defects were those that prevented feature work. Third-party services such as Facebook and Instagram caused around 10% of defects. 44% could be observed in the frontend and 56% in the backend. 38% of all defects were regression bugs.

13% of defects could have been prevented by appropriate unit tests. 72% were in methods with cyclomatic complexity of 3 and above. Cyclomatic complexity of 2 means there is a single conditional statement (if/else). 82% of bugs were introduced in methods with more than 10 lines of code.

This shows that 100% coverage is not needed; getter and setter tests would not have been effective for Emanuil’s team. 36% of bugs could have been prevented by API tests, and 21% could have been caught at the UI level.

The rest 30%?

13 + 36 + 21 does not equal 100 because the remaining 30% could never be detected in the team’s test environment. They could happen only in the real world and were caused by edge cases, configuration issues, unintended software usage, and incomplete requirements.

His advice was to forget this 30%. Focus on early detection and fast recovery.

Customers found 31% of defects in less than one month after they were introduced, and 51% in less than two months. Developers fixed 48% of bugs in one working day and 69% within one working week.

Static code analysis could have caught 2.3% of bugs (SonarQube, ESLint); Emanuil said it was not worth the investment. 6% could have been prevented if Java had been used instead of PHP. Emanuil introduced 5% of the bugs because he made incorrect assumptions about business logic.

Recommendations

Developers should fix their own bugs; this is how they learn. Have at least one test written per method. Do sanity manual checks even for the smallest fixes. Make code reviews mandatory and test with boundary values. Monitoring and fixing exceptions found in production logs is essential. Emanuil’s team monitors exception logs after each automated test execution and considers the run unsuccessful if any exception is present. The team has implemented custom database checks for data quality.

His advice on how to start decreasing customer-reported issues was:

• Allocate time.
• Figure out what you want to track.
• Track customer-reported defects.
• Include the defect ID in the commit message.
• Investigate immediately.

Writing Reliable Web Application UI Tests

Maxim Naidenov started with a typical problem: feedback for developers comes slowly because automation tests take time to execute. Local reproduction can be hard and retesting can be slow. Asking developers to write UI tests is not common practice because UI tests:

• Run in a browser.
• Are flaky.
• May use a different language from the web application.
• Require external tools and complicated local setup.
• Are hard to debug.

Maxim introduced SAPUI5. Its open-source version is called OpenUI5. The difference between SAPUI5 and OpenUI5 is several SAP-specific widgets and the license.

OpenUI5 applications are responsive across browsers and devices. OpenUI5 provides the OPA (One Page Acceptance) UI-testing framework for UI5-based applications. OPA scripts run in the browser and are not executed with Selenium.

An OPA test starts from an HTML page. The script starts and stops the mocked application. A test is a sequence of waitFor calls.

My opinion is that OPA has severe limitations and could be used only for OpenUI5 applications. Read more about the limitations here.

Groom or Doom – Why Junior Talent Is the True Fuel for Your Growth

Svetozar Georgiev shared several years of experience with the IT job market and Telerik Academy students.

HR agencies are like a brokerage service. They do not find new talent; they distribute existing talent among IT companies. Demand is high, and the risks are high compensation and job-hopping tendencies.

The IT-system-friendly way to hire is to train and onboard new people. You can shape them in the way you like. If you motivate and properly mentor juniors, they will advance quickly. They know they do not know much and are eager to learn. You can protect and evolve your company culture. Career switchers are also important. Typically, they have higher motivation for success, lower egos, reasonable expectations, and life maturity. The short-term effort of training and mentoring is worthwhile for long-term growth.

Encourage junior curiosity and allow for “controlled failures.” Juniors learn best when working on real projects, but you should protect both the project and the juniors from their mistakes. Customers should be informed that they are trainees. Ensure juniors get enough attention, stimulate them to learn outside office hours, and give them the bigger picture so they know why they are doing something.

Once you have many seniors, you could move them to another team. Rotate people and build acceptance for change.

Training juniors is a good investment, not an expenditure. Develop new talent instead of stealing it from other companies. This is a sustainable, cost-efficient, controlled, and responsible way to grow.

Running Kubernetes at Scale Across Multiple Cloud Platforms

Iliyan Nenov spoke about Kubernetes. Shipping containers were invented in 1956 and Docker in 2013. Kubernetes is highly efficient, lightweight, cloud-ready, easy to assemble, and has standardized delivery. Kubernetes supports Rocket and Docker portability.

Containers are deployed 900 times faster than virtual machines and utilize the CPU six times better. All containers on a host machine share the same IP. Container-management platforms resolve that challenge; a TCP proxy per container provides an IP per container.

There are more than 60 certified Kubernetes solutions. You can run Kubernetes yourself, use certified Kubernetes distributions, or use managed Kubernetes services such as the Azure public cloud.

Amazon, Google, Microsoft, and Dell invest around 10 billion dollars per year in infrastructure and releasing features. They provide Kubernetes services. Cloud lock-in means there is a huge cost to switch to another cloud service provider.

If you have competitive data and algorithms, they should be kept in-house. Kubernetes has the strongest chance of becoming a standard for cloud portability.

Production Monitoring and Analysis in the Cloud

Nasko Georgiev and Mladen Savov presented their opinions on active and passive application monitoring.

Good performance attracts new customers. Longer load times decrease revenue. Ideally, you can run tests from multiple locations around the world.

Passive monitoring requires actual traffic. You can monitor within the corporate network, with a client-side beacon, and through local instrumentation. Passive monitoring depends on real users, affects business performance, and cannot be measured in pre-production.

Active monitoring simulates user behavior. You can do this with Selenium-based scripts in the cloud. It does not need actual traffic. The scripts cover business needs and scenarios, and do not add load to production systems. Active monitoring covers a limited number of paths and can be useful as part of CI for early detection of issues.

Nasko and Mladen use Neustar. They set a test timeout and measure execution time between test.BeginTransaction and test.endTransaction calls. They use the BigPanda dashboard for smart alerting. If they receive 1,000 alerts for one problem, only one message is sent.

The speakers use two methods for catching performance issues. The first is the three-strike rule: wait for three consecutive functional-test errors or three response times above the set threshold. The second option is to count errors during a period of time.

The conclusion was that it is better to combine active and passive monitoring. You can measure SLA. It is best to compare performance in different locations and browsers.

Making Sense of Big Data through Machine Learning and Statistical Modelling

Dimiter Shalvardjiev presented machine-learning techniques. Probabilities are more efficient than strict rules over large data sets. Data can be labelled, unlabelled, or mixed. For example, a labelled set identifies which samples contain cancer cells and which do not. Experienced doctors were able to predict in 48% of cases whether a patient had cancer; with machine learning, the rate was 76%.

Customer segmentation is an example of an unlabelled data set because there is no binary differentiation. Fraud detection is mixed. One real-life problem that benefits from machine learning is sales maximization: upselling suggestions are based on historical behavior. Spam is no longer working; only 5% of it is effective.

Do not use machine learning for pricing because it is perceived as unfair. Also do not use machine-learning techniques to retarget customers to sell them a different product from the one they initially planned to buy.

Summary

From my point of view, the ISTA conference is mostly suitable for junior to intermediate-level QA engineers. If you expect the presentations to be more technical, you will be disappointed. I did not see preliminary voting for the lectures or speakers. It would also be nice if videos and slides were published right after the event. More than a month after ISTA 2017 ended, they were still unavailable. Here is the ISTA YouTube channel. I hope you find this article useful!