Workshop: Testing #

In this workshop, we will introduce the fundamentals of software testing, and then demonstrate how to write and run tests in Android (unit tests and UI tests).

Testing Basics #

Testing or Quality Assurance (QA) is the software development phase where we test software against its requirements and validate whether it meets stakeholders’ expectations. Testing help you identify and fix bugs early, before they are exposed to users and cause more serious issues. You should start adding tests as soon as you start coding; some argue that tests should be written before the code is written, as executable specifications of the software requirements (Test-Driven Development).

Each test (which is also called a test case) follows three steps:

1. Arrange: prepare the test inputs
2. Act: invoke the code under test
3. Assert (a.k.a. oracles): check the expected outcomes

A collection of test cases is called a test suite.

Categories of Tests #

There are many different kinds of tests that cover different aspects of the software requirements.

By Granularity #

• Unit tests: smaller tests that verify a single method or class in isolation.
• Integration tests: medium-sized tests that check the interaction between several components in the system.
• End-to-end (E2E) tests: larger tests that verify a complete user scenario, usually triggers all components in the system.

The testing pyramid illustrates the granularity of tests. It also suggests that a typical project will have many unit tests, some integration tests, and a few end-to-end tests.

By Subject #

• Functional tests: focus on the business logic and functional requirements
• UI tests: focus on the user interface; usually integration or E2E tests
• Performance tests: focus on checking whether code runs efficiently
• There are also accessibility tests, compatibility tests, etc. targeting other non-functional requirements.

By Methodology #

• Regression tests are most common in testing. They are written down in the codebase, and executed upon each code change to make sure existing functionalities are not accidentally broken. For example, consider a tip calculator function:

// v1 code
fun calculate(amount: Double, tipPercent: Double): Double {
    return tipPercent / 100 * amount
}

// v2 code: added roundUp parameter
fun calculate(amount: Double, tipPercent: Double, roundUp: Boolean = false): Double {
    var tip = tipPercent / 100 * amount
    if (roundUp) { tip = ceil(tip) }
    return tip
}

// regression tests
@Test
fun testCalculateTip() {
    val calculator = TipCalculator()
    calculator.amount = 42.0
    calculator.tipPercent = 10.0

    val tip = calculator.calculateTip()

    assertEquals(4.2, tip, 1e-6)
}

When v2 is released, existing regression tests (e.g., testCalculateTip()) should still pass. If they fail, there are two possible reasons: the code change introduced some bugs -> fix code; the change in the behavior is expected -> update test. New regression tests (e.g., testCalculateTipWithRoundUp()) should also be added to cover the new functionality.

• Random tests (fuzzing): execute the code under test with a lot of random test inputs, where the test oracle is usually not crashing or not violating certain properties/invariants. The goal is to automatically find bugs with less human effort to curate test cases.
• Differential tests: some software (e.g., compilers) are designed to execute in multiple configurations (e.g., with optimization flags vs. without optimization flags), where the execution results are expected to be same/similar (e.g., the compiled code should have the same functionality despite efficiency differences). A differential test gives the same input to the two configurations and checks whether the results are same/similar.
• Metamorphic tests: some software (e.g., stop sign detector) has a large space for possible inputs, where small variations in a input (e.g., small noise added to the image) should not cause significant changes in the output (e.g., the stop sign is still detected). A metamorphic test checks relationships between outputs for a set of related inputs; test oracle is a postcondition derived from the relationship between inputs.

Code Coverage #

Code coverage is a quality metric for your test suites. It measures what percentage of code elements are covered (executed) during tests. A high-quality codebase should target 100% coverage.

There are two types of coverage metrics that are commonly used:

• Line coverage: the percentage of lines covered.
• Branch coverage: the percentage of branches (including conditional statements, switch statements, loops, etc.) covered.

Testing in Android #

For the scope of Android app development, the following two types of regression tests are frequently used

• Unit tests
  • directory: src/test/java
  • documentation: Build local unit tests, JUnit 5
• UI tests (at the E2E tests level)
  • directory: src/androidTest/java
  • documentation: Build instrumented tests, Jetpack Compose Testing

Demo #

The demo we will be following is a Tip Calculator app. You can clone the code and open it in Android Studio to follow along.

The code under test is a TipCalculator class that computes the tip for a bill:

class TipCalculator(
    var amount: Double = 0.0,
    var tipPercent: Double = 0.0,
    var roundUp: Boolean = false,
) {
    fun calculateTip(): Double {
        var tip = tipPercent / 100 * amount
        if (roundUp) {
            tip = ceil(tip)
        }
        return tip
    }
}

Writing and Running Unit Tests #

Unit tests live in src/test/java. We will write two test suites here (in Java/Kotlin, a class ends with Test is usually considered as a test suite):

• TipCalculatorTest: for testing the TipCalculator class, which is the model layer in the MVVM architecture. This test suite checks whether the business logic is correctly implemented.
• TipTimeViewModelTest: for testing the TipTimeViewModel class, which is the ViewModel layer in the MVVM architecture. This test suite checks whether the UI state is correctly updated (e.g., here we test if the tip amount is correctly calculated and formatted as string).

To run the unit tests, create a run configuration in Android Studio (in the background Gradle will be used to run the tests):

Once you click run, the tests will be executed and the results will be shown in the Run window below. You can explore the list of tests that passed and failed (by default only the failed ones are shown in the list, but you can toggle it by clicking the small tick icon “Show Passed”).

You can also run unit tests with coverage from the extra menu “Run with Coverage”. This will generate a summary coverage report (to the right), and coverage-based highlights in the code editor to help you identify which lines are not covered.

Writing and Running UI Tests #

UI tests live in src/androidTest/java. They use the Jetpack Compose testing library to interact with UI elements.

In our demo, the TipTimeUITests suite has one test case calculate_20_percent_tip() that: fills in the text fields with bill amount and tip percentage, and then checks if the tip amount (with labels and formatting) is correctly displayed on the screen.

To run the UI tests, create a run configuration in Android Studio as “Android instrumented tests”:

UI tests run on a emulator or physical device. You will see the UI interactions being automatically performed by the testing framework.