# Blackfire Player

Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to **crawl HTTP services**, **assert responses**, and **extract data** from HTML/XML/JSON responses.

Some Blackfire Player use cases:

- Crawl an application/API and check expectations -- aka Acceptance Tests;
- Scrape an application/API and extract values;
- Synthetic monitoring of an application;
- Test code with unit test integration (PHPUnit, Behat, Codeception, ...);
- Test code behavior from the outside thanks to the native Blackfire Profiler integration -- aka *Unit Tests from the HTTP layer* (tm).

> **Note**
>
> Blackfire Player executes scenarios written in a special DSL described in the `blackfire.yaml` file or in files ending with `.bkf`.
>
> Use the [online validator](https://blackfire.io/validator)to validate the syntax of your tests files.

## Usage

`blackfire-player` is distributed through a Docker image.

To run a scenario located in the `my-scenario.bkf` file, run the following command:

The `pwd` part is the local working directory (we are using the current directory here) and it is mapped to the `/app` path in the Docker container.

`BLACKFIRE_CLIENT_ID` and `BLACKFIRE_CLIENT_TOKEN` environment variables need to be properly exposed from the host in order to be able to use the [Blackfire Profiler integration](../integrations/blackfire-player).

> **Note**
>
> To make it simpler to run this command, you might create a shell alias (that you can store in a `.bashrc` or `.zshrc` file depending on your shell):
>
> Don't forget to restart your terminal for it to take effect. You can then use `blackfire-player` as if it was the binary itself:
>
> ```bash
> blackfire-player --version
> blackfire-player list
> blackfire-player run my-scenario.bkf
> ```

### Use Blackfire Player from another Docker container

Using the `host` network mode for the `blackfire/player` container allows to bypass the isolation of containers and permits Blackfire Player to trigger requests on your application container.

The `--endpoint` option should be set to your application container public URL. Depending on your configuration, you might need the `--ssl-no-verify` option to mitigate eventual local certificate issues.

```bash
docker run --rm -it \
    --network host \
    -e BLACKFIRE_CLIENT_ID \
    -e BLACKFIRE_CLIENT_TOKEN \
    -v "`pwd`:/app" \
    blackfire/player run my-scenario.bkf \
    --endpoint <CONTAINER_PUBLIC_URL> \
    --ssl-no-verify
```

`DDEV` integration documentation provides a [convenient custom command](../integrations/paas/ddev#ddev-player).

## Blackfire Player commands

### The "run" Command

Use the `run` command to execute a scenario file:

```bash
blackfire-player run scenario.bkf
```

> **Note**
>
> The *file* argument may be omitted when using the standard input:
>
> ```bash
> cat scenario.bkf | blackfire-player run
> ```

You can also run scenarios contained in a `.blackfire.yaml` file:

```bash
blackfire-player run .blackfire.yaml
```

Use the `--endpoint` option to override the endpoint defined in the scenario file:

```bash
blackfire-player run scenario.bkf --endpoint=http://example.com/
```

Use the `--json` option to output a JSON report:

```bash
blackfire-player run scenario.bkf --json
```

Use the `--variable` option to override variable values:

```bash
blackfire-player run scenario.bkf --variable "foo=bar" --variable "bar=foo"
```

Use the `--concurrency` option to run scenarios in parallel (experimental):

```bash
blackfire-player run scenario.bkf --concurrency=5
```

Use `-v` to get logs about the progress of the player or use `tracer` option to store all requests and responses on disk.

Use the `--blackfire-env` option to define the [environment](../reference-guide/environments)to which the profiles are sent. It is required only when using the [Blackfire Profiler integration](../integrations/blackfire-player).

```bash
blackfire-player run my-scenario.bkf --blackfire-env=<ENVIRONMENT_NAME_OR_UUID>
```

Use the `--report` option to print an aggregated report after all scenarios complete. It lists each profiled step with a link to its profile and key performance figures. Requires `--blackfire-env`:

```bash
blackfire-player run my-scenario.bkf --blackfire-env=<ENVIRONMENT_NAME_OR_UUID> --report
```

> **Note**
>
> `--report` is the recommended option when running [synthetic monitoring](synthetic-monitoring)from a CI/CD pipeline or deployment script.

**The "run" command uses the following exit codes in case of failure:**

- `64` if at least one scenario fails;
- `65` if a fatal error occurs, preventing the scenarios from running;
- `66` if a non-fatal error occurs;

### The "validate" Command

The `validate` command checks if passed scenario file is valid:

```bash
blackfire-player validate scenario.bkf
```

The *file* argument may be omitted when using the standard input:

```bash
cat scenario.bkf | blackfire-player validate
```

> **Note**
>
> It is not possible to validate a scenario contained in a `.blackfire.yaml`file.

Use the `--json` option to output a JSON report:

```bash
blackfire-player validate scenario.bkf --json
```

**The command uses the following exit codes in case of failure:**

- `64` if the file is invalid;

## Crawling an HTTP application

Blackfire Player lets you crawl an application thanks to descriptive scenarios written in a domain specific language:

```blackfire
name "A set of scenarios"

# Default endpoint
# Can be overridden with the "--endpoint=http://newendpoint.com" option
endpoint "http://example.com/"

scenario
    name "Scenario Name"

    visit url('/')
        expect status_code() == 200
```

This example shows how to make a request on an HTTP application (`http://example.com/`) and be sure that it behaves the way you expect it to by Writing Expectations (the status code of the response is 200).

Store the scenario in a `scenario.bkf`, and run it:

```bash
blackfire-player run scenario.bkf

# or
php blackfire-player run scenario.bkf
```

Add more requests to a scenario by indenting lines as below:

```blackfire
scenario
    visit url('/')
        expect status_code() == 200

    visit url('/blog/')
        expect status_code() == 200
```

> **Note**
>
> The line indentation defines the structure like for Python scripts or YAML files. Validate `bkf` files with the `validate` command: `blackfire-player validate scenario.bkf`.

A **scenario** is a sequence of HTTP calls (**steps**) that share the HTTP session and cookies. Scenario definitions are **declarative**, the order of settings (like expectations) within a "step" does not matter.

Instead of making discrete requests like above, you can also **interact** with the HTTP response if the content type is HTML by clicking on links, submitting forms, or follow redirections (see [Making requests](player#making-requests) for more information):

```blackfire
scenario
    visit url('/')
        expect status_code() == 200

    click link('Read more')
        expect status_code() == 200
```

> **Note**
>
> If your scenario does not work as expected, use `-v` to get a more verbose output.

> **Tip**
>
> You can add comments in a scenario file by prefixing the line with `#`:
>
> ```blackfire
> # This is a comment
> scenario
>     # Comment are ignored
>     visit url('/')
>         expect status_code() == 200
> ```

### Making Requests

There are several ways you can jump from one HTTP request to the next.

#### Visiting a Page with `visit`

`visit` goes directly to the referenced HTTP URL (defaults to the `GET`HTTP method unless you define one explicitly):

```blackfire
scenario
    visit url('/')
        method 'POST'
```

You can also pass a Request body:

```blackfire
scenario
    visit url('/')
        method 'PUT'
        body '{ "title": "New Title" }'
```

> **Tip**
>
> An expression can be written on several lines with the following syntax:
>
> ```blackfire
> scenario
>     visit url('/login')
>         method 'POST'
>         body
>         """
>         {
>             "user": "john",
>             "password": "doe"
>         }
>         """
> ```
>
>
>
> Starting from version v1.11.0 you can also use variables by adding `i` option to multiline string.
>
> ```blackfire
> scenario
>     visit url('/login')
>         method 'POST'
>         set username "john"
>         set password "doe"
>         body
>         """i
>         {
>             "user": "${username}",
>             "password": "${password}"
>         }
>         """
> ```

#### Clicking on a Link with `click`

`click` clicks on a link in an HTML page (takes an expression as an argument):

```blackfire
scenario
    click link("Add a blog post")
```

#### Submitting Forms with `submit`

`submit` submits a form in an HTML page (takes an expression as an argument); parameters to submit with the form are defined via `param` entries:

```blackfire
scenario
    submit button("Submit")
        param title 'Happy Scraping'
        param content 'Scraping with Blackfire Player is so easy!'

        # File Upload:
        # the path is relative to the current .bkf file
        # the name parameter is optional
        param image file('relative/path/to/image.png', 'blackfire.png')
```

Values can also be randomly generated via the `fake()` function:

```blackfire
scenario
    submit button("Submit")
        param title fake('sentence', 5)
        param content join(fake('paragraphs', 3), "\n\n")
```

Generate random images with the `simple_image` generator:

```blackfire
scenario
    submit button("Submit")
        param image file(fake('simple_image', null, 400, 300, 'png', true, true), 'placeholder.png')
```

> **Note**
>
> `fake()` use the [Faker library](https://github.com/FakerPHP/Faker)under the hood.

#### Following Redirections

HTTP redirections are never followed automatically to let you write expectations and assertions on redirect responses:

```blackfire
scenario
    visit "redirect.php"
        expect status_code() == 302
        expect header('Location') == '/redirected.php'
```

Use `follow` to follow one redirection:

```blackfire
scenario
    visit "redirect.php"
        expect status_code() == 302
        expect header('Location') == '/redirected.php'

    follow
        expect status_code() == 200
```

`follow_redirects` switches the player to automatically follow all redirections:

```blackfire
scenario
    follow_redirects true
```

or:

```blackfire
scenario
    visit "redirect.php"
        follow_redirects
```

Please note that when using `follow_redirects`, expectations (`expect`) and assertions (`assert`) are checked on the redirecting response (so, before the redirection). Use a `follow` step if you need to check them after the redirection.

#### Embedding Scenarios with `include`

`include` allows to embed some repetitive steps into several scenarios to avoid copy/pasting the same code over and over again:

In a `groups.bkf` file, write a `group` that contains the logic to log in:

```blackfire
group login
    visit url('/login')
        expect status_code() == 200

    submit button('Login')
        param user 'admin'
        param password 'admin'
```

Then, in another file, `load` the `group` and `include` it when you need it:

```blackfire
load "groups.bkf"

scenario
    name "Scenario Name"

    include login

    visit url('/admin')
        expect status_code() == 200
```

### Configuring the Request

Each step can be configured via the following options.

#### Setting a Header with `header`

`header` sets a header:

```blackfire
scenario
    visit url('/')
        header "Accept-Language: en-US"
```

> **Tip**
>
> Simulate a specific browser is as simple as overriding the default `User-Agent` and using `fake()`:
>
> ```blackfire
> scenario
>     visit url('/')
>         header 'User-Agent: ' ~ fake('firefox')
> ```

#### Setting a User and Password with `auth`

`auth` sets the `Authorization` header:

```blackfire
scenario
    visit url('/')
        auth "username:password"
```

#### Waiting after sending the request with `wait`

`wait` adds a delay in milliseconds after sending the request:

```blackfire
scenario
    visit url('/')
        wait 10000
```

The `wait` value can be any valid expression; get a random delay by using `fake()`:

```blackfire
scenario
    visit url('/')
        wait fake('numberBetween', 1000, 3000)
```

#### Sending a JSON Body with `json`

`json` configures the Request to upload JSON encoded data as the body:

```blackfire
scenario
    visit url('/')
        method 'POST'
        param foo "bar"
        json true
```

#### Setting Options for all Steps

You can also set some of these options for all steps of a scenario:

```blackfire
scenario
    auth "username:password"
    header "Accept-Language: en-US"
```

... which can be disabled on any given step by setting the value to `false`:

```blackfire
scenario
    visit url('/')
        header "Accept-Language: false"
        auth false
```

## Writing Expectations

Expectations are **expressions** evaluated against the current HTTP response and if one of them returns a *falsy* value, Blackfire Player stops the run and generates an error.

Expressions have access to the following functions:

- `current_url()`: Returns the current URL
- `status_code()`: The HTTP status code for the current HTTP response;
- `header()`: Returns the value of an HTTP header;
- `body()`: The HTTP body for the current HTTP response;
- `trim()`: Strip whitespace from the beginning and end of a string;
- `unique()`: Removes duplicate values from an array;
- `join()`: Join array elements with a string;
- `merge()`: Merge one or more arrays;
- `regex()`: Perform a regular expression match;
- `css()`: Returns nodes matching the CSS selector (for HTML responses);
- `xpath()`: Returns nodes matching the XPath selector (for HTML and XML responses);
- `json()`: Returns JSON elements (from the request) matching the CSS expression.
- `transform()`: Returns JSON elements matching the CSS expression.

The `css()` and `xpath()` functions return `Symfony\Component\DomCrawler\Crawler` instances. Learn more about [methods you can call on Crawler instances](http://symfony.com/doc/current/components/dom_crawler.html); the `json()`function returns a PHP array.

The `json()` function accepts [JMESPath](http://jmespath.org/specification.html).

The result of calling functions can be checked via [operators](http://symfony.com/doc/current/components/expression_language/syntax.html#supported-operators) described.

> **Note**
>
> Learn more about [Expressions syntax](http://symfony.com/doc/current/components/expression_language/syntax.html)in the Symfony documentation.

Here are some expression examples:

```blackfire
# return all HTML nodes matching ".post h2 a"
css(".post h2 a")

# return the text of the first node matching ".post h2 a"
css(".post h2 a").first().text()

# return the href attribute of the first node matching ".post h2 a"
css(".post h2 a").first().attr("href")

# check that "h1" contains "Welcome"
css("h1:contains('Welcome')").count() > 0

# same as above
css("h1").first().text() matches "/Welcome/"

# return the Age request HTTP header
header("Age")

# check that the HTML body contains "Welcome"
body() matches "/Welcome/"

# get a value
json("_links.store.href")

# get keys
json("arguments."sql.pdo.queries".keys(@)")
```

## Using Variables

Variables can be defined to make your scenarios dynamic. Use `set` to define the default value:

```blackfire
scenario
    name "HTTP Cache"
    set env "dev"

    when "prod" == env
        visit url('/')
        # check HTTP cache, but only on production
```

And override it with the `--variable` option on the CLI:

```bash
blackfire-player run scenario.bkf --variable env=prod
```

## Iterating

Use `with` to iterate over a set of data:

```blackfire
scenario
    name "HTTP Cache"
    set paths ["/", "/blog/"]

    with path in paths
        visit url(path)
            name "Checking performance on path: " ~ path
            expect status_code() == 200
            # performance checks

scenario
    name "Checks on key pages"

    with name, data in \
        { \
            admin: { slug: "/admin/", expectedStatusCode: 401 }, \
            products: { slug: "/products/", expectedStatusCode: 200 }, \
            about: { slug: "/about/", expectedStatusCode: 200 } \
        }

        visit url(data["slug"])
            name "Checking performance on path: " ~ name
            expect status_code() == data["expectedStatusCode"]
```

## Loops

Use `while` to perform loops:

```blackfire
scenario
    name "While loops"
    visit url('/products/')
    set pageCount css(".max_results_count").first().text()

    set page 1
    while page < pageCount
        visit url('/products/?page=' ~ page)
            set page page + 1
            expect status_code() == 200
            # performance checks
```

## Organizing Scenario Files

To run scenarios defined in several files, you can use `load` instead of listing all the files as arguments to the player:

```blackfire
# load and execute all scenarios from files in this directory
load "*.bkf"

# load and execute all scenarios from files in all sub-directories
load "**/*.bkf"
```

## Blackfire Profiler integration

Blackfire Player integrates seamlessly with Blackfire Profiler. Read out [the dedicated documentation to learn more about Blackfire Profiler integration](https://docs.blackfire.io/integrations/blackfire-player).

## Scraping Values

When crawling an HTTP application, you can extract values from HTTP responses:

```blackfire
scenario
    visit url('/')
        expect status_code() == 200
        set latest_post_title css(".post h2").first()
        set latest_post_href css(".post h2 a").first().attr("href")
        set latest_posts css(".post h2 a").extract('_text', 'href')
        set age header("Age")
        set content_type header("Content-Type")
        set token regex('/name="_token" value="([^"]+)"/')
```

`set` takes two arguments:

- The name of the variable you want to store the value in;
- An expression to evaluate.

Using `json()`, `css()`, and `xpath()` on JSON, HTML, and XML responses is recommended, but for pure text responses or complex values, you can use the generic `regex()` function.

> **Note**
>
> `regex()` takes a regex as an argument and always returns the first captured parenthesized subpattern. Note that backslashes must be escaped by doubling them: `"/\\.git/"`.

The values are also available at the end of a crawling session:

```bash
# use --json to display a report including variable values
blackfire-player run scenario.bkf --json
```

Variable values can also be injected before running another scenario:

```blackfire
scenario
    name "Scenario name"
    auth api_username ~ ':' ~ api_password
    set profile_uuid 'zzzz'

    visit url('/profiles' ~ profile_uuid)
        expect status_code() == 200
        set sql_queries json('arguments."sql.pdo.queries".keys(@)')
        set store_url json("_links.store.href")

    visit url(store_url)
        method 'POST'
        body '{ "foo": "batman" }'
        expect status_code() == 200
```