Black Box Testing

Introduction

Black box testing, also known as behavioural testing, is a type of software testing that focuses on testing a system’s functionality and behaviour without considering its internal structure or implementation details. In other words, it treats the system as a “black box” and tests its inputs and outputs to ensure it behaves correctly according to the specified requirements.

Black box testing can be performed at various levels of the software development process, including unit testing, integration testing, system testing, and acceptance testing. It typically involves creating test cases based on the system’s functional specifications or user stories without considering how the system is implemented. These test cases are then executed to verify that the system behaves as expected for a given set of inputs.

This article aims to provide a comprehensive understanding of black box testing, its importance, and its application in API development. Since the focus is entirely on understanding black box testing as a concept and not the API development process, Python is an ideally suited language for the demo as it is simple and easy to understand.

Advantages

Black box testing has several advantages over other types of testing:

• Black box testing helps ensure that the system meets its specified requirements and provides value to its users.

• Black box testing reduces the risk of introducing bugs or defects during development, as it focuses on the system’s external behaviour rather than its internal implementation.

• Black box testing can be performed by testers who are unfamiliar with the system’s implementation details. This user-centric approach ensures that the system is easy to use and understand for a wide range of users, enhancing its usability and accessibility.

• Black box testing fosters a spirit of collaboration between developers and testers. Both groups must work together to create compelling test cases and ensure the system meets its requirements. This collaborative approach not only improves the system’s quality but also enhances the overall development process.

Limitations

However, black box testing also has some limitations:

• It may not be able to detect specific bugs or defects related to the system’s internal implementation or architecture.

• Creating practical test cases that cover a wide range of scenarios and inputs can be time-consuming and labour-intensive.

• Creating and executing test cases, particularly for complex systems or applications, may require specialized tools or expertise.

Black Box Testing in practice

In the previous blog post on Test-Driven development, we discussed how to create unit tests and integration tests for an API endpoint that calculates the area of a circle based on its radius. The integration tests perform HTTP requests against the API by interacting with it as a user would, verifying our assumptions on the responses we receive. This is often referred to as black box testing.

API endpoint:

from flask import Flask, jsonify, request
import math

app = Flask(__name__)

def circle_area(radius):
    return math.pi * radius ** 2

@app.route('/circle-area')
def get_circle_area():
    try:
        radius = float(request.args.get('radius'))
        if radius <= 0:
            return {'error': "Radius must be a positive number."}, 400
    except ValueError:
        return {'error': "Radius must be a positive number."}, 400
    area = circle_area(radius)
    return {'area': area}

if __name__ == '__main__':
    app.run()

Writing Tests

Black box testing focuses on testing the external behaviour of a system without considering its internal structure or implementation details. In our case, we want to test the /circle-area endpoint of our API to ensure that it correctly calculates and returns the area of a circle based on a given radius parameter. The complete code for the demo can be found in the repo here. In the black box tests, the URL http://localhost:5000 is deliberately used instead of creating Flask’s test_client(). This is to completely decouple the black box tests from the integration tests. It shows that although black box tests and integration tests are almost always used interchangeably, black box tests can be run entirely independently by replacing the localhost URL with the dev/test system’s deployed URL.

Here are some examples of black-box integration tests for this endpoint:

Test No. 1: Testing with a valid positive radius

We can create a test case where we send a GET request to /circle-area?radius=5 and expect a response with an HTTP status code of 200 and a JSON payload containing the area of the circle.

import json
import requests
import math

def test_circle_area_valid_positive_radius():
	"""Test the /circle-area endpoint with a valid positive radius"""

	response = requests.get("http://localhost:5000/circle-area?radius=5")
	assert response.status_code == 200
	result = json.loads(response.content)
	assert "area" in result
	assert round(result["area"], 4) == round(math.pi * (5**2), 4)

Test No. 2: Testing with a zero or negative radius

We can create another test case where we send a GET request to /circle-area?radius=0 or /circle-area?radius=-5 and expect a response with an HTTP status code of 400 and an error message indicating that the radius parameter must be a positive number.

import json
import requests
import math
from parameterized import parameterized

@parameterized.expand([
        (0, "Radius must be a positive number."),
        (-20, "Radius must be a positive number.")
    ])
def test_circle_area_zero_or_negative_radius(radius, expected):
    """Test the /circle-area endpoint with a zero or negative radius"""
    response = requests.get("http://localhost:5000/circle-area?radius={}".format(radius))
    assert response.status_code == 400
    result = json.loads(response.content)
    assert "error" in result
    assert result["error"] == expected

These types of tests are called Parameterised Tests, explained in this section of the TDD.

Test No. 3: Testing with non-numeric input

We can create another test case where we send a GET request to /circle-area?radius=hello or /circle-area?radius=5.5f and expect a response with an HTTP status code of 400 and an error message indicating that the radius parameter must be a valid number.

import json
import requests
import math

def test_circle_area_nonnumeric_radius():
    """Test the /circle-area endpoint with non-numeric input"""

	response = requests.get("http://localhost:5000/circle-area?radius=hello")
	assert response.status_code == 400
	result = json.loads(response.content)
	assert "error" in result
	assert result["error"] == "Radius must be a positive number."

Test No. 4: Testing with massive input

We can create another test case where we send a GET request to /circle-area?radius=1e6 or /circle-area?radius=1000000 and expect a response with an HTTP status code of 200 and the correct area value, even for enormous radii.

import json
import requests
import math

def test_circle_area_large_radius():
    """Test the /circle-area endpoint with very large input"""

	response = requests.get("http://localhost:5000/circle-area?radius=1e6")
	assert response.status_code == 200
	result = json.loads(response.content)
	assert "area" in result
	assert round(result["area"], 4) == round(math.pi * (1e6**2), 4)

Test No. 5: Testing for rounding errors

We can create another test case where we send a GET request to /circle-area?radius=0.1 or /circle-area?radius=0.001 and expect a response with an HTTP status code of 200 and the correct area value, rounded to the nearest float value (e.g., 0.0314 for radius 0.1).

import json
import requests
import math

def test_circle_area_rounding():
    """Test the /circle-area endpoint for correct rounding of small numbers"""

	response = requests.get("http://localhost:5000/circle-area?radius=0.1")
	assert response.status_code == 200
	result = json.loads(response.content)
	assert "area" in result
	assert round(result["area"], 4) == round(math.pi * (0.1**2), 4)

These test cases assume your API is running on localhost:5000. If it’s running on a different host or port, or you are testing an endpoint deployed to a dev/test server, you can modify the URL accordingly. Also, note that these tests use the Pytest framework for testing and the requests module to make HTTP requests to your API. If you’re using a different testing framework or don’t have the requests module installed, you may need to modify this code accordingly.

Bottom Line

These test cases are examples of black box integration tests because they focus on testing the external behaviour of the /circle-area endpoint without considering its internal implementation details or structure. We do not need to know how the endpoint calculates the area value as long as we can verify that it returns the correct value for a given set of inputs and that it handles invalid inputs correctly. Using a black box testing approach, we can ensure that the /circle-area endpoint meets its specified requirements and behaves as expected in different scenarios and use cases.

Conclusion

Overall, black box testing is an essential part of the software development process, as it helps ensure that the system meets its specified requirements and provides value to its users. It allows for the validation of software functionality without needing to understand the underlying code or structure. By focusing on the system’s external behaviour rather than its internal implementation, black box testing can help identify defects and bugs early in the development process, reduce maintenance costs, and improve overall product quality.