Loading...
  1. Documentation
  2. Testing Cookbooks
  3. Writing Tests

Writing Tests
Requires Production Plan

Blackfire gathers a lot of data about how your code behaves at runtime, from wall-clock time, to I/O time or CPU time, memory, number of calls to a given function, number of SQL queries and their execution time, and many others. Some of these metrics are built-in, and you can create your own as well.

Pretty much like writing unit tests or integration tests, Blackfire enables you to write tests with these metrics. These can be performance tests, which help you make sure your code sticks to a performance budget you have defined. And if needed, such tests can go beyond performance, like our quality and security recommendations.

Tests will be run on any profile you generate, may it be on-demand (like via the browser or the CLI), or automatically (like via the Blackfire Player or Builds).

Test results will be displayed via the profile view (under the assertions tab on the left-hand side), or aggregated in a Build report view.

The best way to get started with writing Blackfire tests is via the .blackfire.yaml file.

The .blackfire.yaml file is a YAML file where tests, metrics, and scenarios can be defined. It must be part of your code repository:

1
2
3
4
5
6
7
8
9
10
tests:
    "Pages should be fast enough":
        path: "/.*" # run the assertions for all HTTP requests
        assertions:
            - "main.wall_time < 100ms" # wall clock time is less than 100ms

    "Commands should be fast enough":
        command: ".*" # run the assertions for all CLI commands
        assertions:
            - "main.wall_time < 2s" # wall clock time is less than 2s

To get started, create a .blackfire.yaml file in the root directory of an application. You can bootstrap a new .blackfire.yaml file with the following command: blackfire client:bootstrap-tests.

Use the online .blackfire.yaml validator to validate the syntax of your .blackfire.yaml files.

Now that you have created the file, read the following paragraphs to learn more about how to write your tests.

A test must be located under the tests key and is composed of the following required items:

  • A name (e.g. "Pages should be fast enough");

  • A context. It can be either an HTTP request or a CLI command.

    • An HTTP request is identified with a regular expression on its path, under the path key, where you may specify one path or a collection of paths. You may specify HTTP methods under the methods key. It is possible to exclude one or several paths by specifying them under the exclude key;
    • A CLI command is identified with a regular expression on the command itself, under the command key.
  • A set of assertions. Read the assertion reference guide to learn more about the Blackfire assertion syntax;
  • An optional description. Learn more about the description.

Here is an example with several assertions limited to some API calls of the application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
tests:
    "Homepage should not hit the DB":
        path:
            - '^/api/article/\d+$'
            - '^/api/category/[^/]+/details$'
        exclude:
            - '^/api/category/all/details'
        methods: [GET, HEAD]
        assertions:
            # no SQL statements executed
            - "metrics.sql.queries.count == 0"
            # memory does not exceed 10mb
            - "main.peak_memory < 10mb"
            # the response size as generated by the instrumented language is less than 100kb
            - "metrics.output.network_out < 100kb"
        description: |
            Optional information explaining the reason of this test or provide
            educational content helping other team members or future self fix it.

            The description could be multiline.

When a profile is made on a project that contains a .blackfire.yaml file, Blackfire automatically runs all tests matching the HTTP request path. The result of the tests is displayed as a green or red icon in the dashboard and the full report is available on the profile page. The same goes when profiling a CLI script via blackfire run.

Note that assertions in the report contain the actual metric and variable values so that you know if you are close to the target or not (metrics.sql.queries.count 5 == 0; 0 is the target, 5 is the actual number of SQL statements executed).

A condition is an expression similar to an assertion. If the condition is fulfilled, the test is evaluated.

There are two kinds of conditions: when and unless. They can be used separately or combined for complex conditions.

A when expression acts like an if condition. The corresponding test is evaluated if the expression returns true.

1
2
3
4
5
6
tests:
    'A database connection should be opened only when queries are made':
        path: '/.*'
        when: "metrics.sql.connections.count > 0"
        assertions:
            - 'metrics.sql.queries.count > 0'

The example above means "For any HTTP request, if a database connection is used, at least 1 SQL query must be run".

An unless expression acts like an if not condition. The corresponding test is evaluated unless the expression returns true.

1
2
3
4
5
6
tests:
    'Twig template cache should be enabled in production':
        path: '/.*'
        unless: 'is_dev()'
        assertions:
            - 'metrics.twig.compile.count == 0'

The example above means "For any HTTP request, unless the profile is run in a development environment, Twig template path should be enabled".

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
tests:

    assertion_label:
        # A path or an array of paths, when in a web context;
        path:
            # Regular expression the web path must match to trigger the assertions.
            - '^/api/article/\d+$'
            - '^/api/category/[^/]+/details$'

        # A path or an array of paths to exclude, when in a web context;
        exclude:
            - '^/foo/bar'

        # A command, when in CLI context;
        # Regular expression the command must match to trigger the assertions.
        command: bin/console project:cron

        # A collection of HTTP Methods.
        # Use ANY for accepting anything (it's the default value).
        methods: # Example: [ GET, POST ]
            # Default:
            - ANY

        # A condition. If not valid, the test will not be used.
        when:                 ~ # Example: is_dev() == true

        # A condition. If valid, the test will not be used.
        unless:               ~ # Example: is_dev() == true

        # A collection of assertions
        assertions:
            -
                label:                ~ # Example: No more than 5 SQL query
                expression:           ~ # Required, Example: metrics.sql.queries.count < 5

        # An optional description
        description: |
            Some optional information on your test