How to add standard tests to a tool

When creating either a custom tool or a new tool to publish in a LangChain integration, it is important to add standard tests to ensure the tool works as expected. This guide will show you how to add standard tests to a tool.

Setup

First, let's install 2 dependencies:

• langchain-core will define the interfaces we want to import to define our custom tool.
• langchain-standard-tests (installed as a git dependency) will provide the standard tests we want to use.

note

The langchain-standard-tests package is not released to pypi, so we need to install it from the git repository.

This will always pull in the latest version of the package.

%pip install -U langchain-core "git+https://github.com/langchain-ai/langchain.git#subdirectory=libs/standard-tests"

Let's start with the custom tool defined in the first part of how to create tools.

app/tools.py

from langchain_core.tools import tool


@tool
def multiply(a: int, b: int) -> int:
    """Multiply two numbers."""
    return a * b

This will sit alongside our test files in the following layout:

/
├── app/
│   └── custom_tool.py
└── tests/
    ├── unit_tests/
    │   └── test_custom_tool.py
    └── integration_tests/
        └── test_custom_tool.py

Add and configure standard tests

There are 2 namespaces in the langchain-standard-tests package: unit tests (langchain_standard_tests.unit_tests) and integration tests (langchain_standard_tests.integration_tests).

Unit tests are designed to be used to test the tool in isolation and without access to external services.

Integration tests are designed to be used to test the tool with access to external services (in particular, the external service that the tool is designed to interact with).

note

Integration tests can also be run without access to external services, if they are properly mocked.

Both types of tests are implemented as pytest class-based test suites.

By subclassing the base classes for each type of standard test, you get all of the standard tests for that type, and you can override the properties that the test suite uses to configure the tests.

Standard tools tests

Here's how you would configure the standard unit tests for the custom tool, e.g. in tests/test_custom_tool.py:

tests/unit_tests/test_custom_tool.py

from typing import Type

from app.tools import multiply
from langchain_standard_tests.unit_tests import ToolsUnitTests


class MultiplyToolUnitTests(ToolsUnitTests):
    @property
    def tool_constructor(self) -> Type[multiply]:
        return multiply

    def tool_constructor_params(self) -> dict:
        # if your tool constructor instead required initialization arguments like
        # `def __init__(self, some_arg: int):`, you would return those here
        # as a dictionary, e.g.: `return {'some_arg': 42}`
        return {}

    def tool_invoke_params_example(self) -> dict:
        """
        Returns a dictionary representing the "args" of an example tool call.

        This should NOT be a ToolCall dict - i.e. it should not
        have {"name", "id", "args"} keys.
        """
        return {"a": 2, "b": 3}

tests/integration_tests/test_custom_tool.py

from typing import Type

from app.tools import multiply
from langchain_standard_tests.integration_tests import ToolsIntegrationTests


class MultiplyToolIntegrationTests(ToolsIntegrationTests):
    @property
    def tool_constructor(self) -> Type[multiply]:
        return multiply

    def tool_constructor_params(self) -> dict:
        # if your tool constructor instead required initialization arguments like
        # `def __init__(self, some_arg: int):`, you would return those here
        # as a dictionary, e.g.: `return {'some_arg': 42}`
        return {}

    def tool_invoke_params_example(self) -> dict:
        """
        Returns a dictionary representing the "args" of an example tool call.

        This should NOT be a ToolCall dict - i.e. it should not
        have {"name", "id", "args"} keys.
        """
        return {"a": 2, "b": 3}

and you could run these with the following commands

# run unit tests
pytest tests/unit_tests

# run integration tests
pytest tests/integration_tests

Advanced unit test running

In all of our official CI, we run the tests with the following pytest plugins in order to enforce no network access.

pip install -U pytest-socket
pytest --disable-socket --enable-unix-socket tests/unit_tests

Integration tests are run without these flags because we allow network access.

pytest tests/integration_tests