Feedback from Cypress tests

A few months ago I mentioned a first unsuccessful attempt to test Cypress with Drupal.

I've since tried again and managed to use Cypress in a Docker environment.

In order to share the code, I created the following snippet : https://gitlab.com/-/snippets/2210867

Using with Docker and Docker Compose

In order to avoid having to deal with dependency issues and having to weigh down my development Docker images or maintain new ones, I wanted to use the official Cypress Docker images.

After various trials, I used the "cypress/included" image to simplify use as the image already embeds everything you need without having anything extra to install.

The problem was that the image is intended to run the tests and then destroy the container.

Or I'm in a Docker Compose environment and I want to be able to use the image as for Pa11y tests, with a Make command that executes a Docker Compose command so that the container is in the project network.

So the image is waiting for a command, and without a command the container is not properly instantiated with Docker Compose. The trick was to put a default command "--help", so the return is not in error and it is then possible to run "docker-compose run" commands on the container.

I then created various Makefile commands to run the tests with different browsers.

Limitations and difficulties encountered

The main difficulty when writing the tests was understanding how Cypress traverses and manipulates DOM elements, as for example some assertions cause the current element to be changed, other assertions do not.

This point is more of a habit to get used to than a difficulty.

User login

As with Behat tests, the Antibot module caused me problems because the examples found for logging in consisted of a POST request to the URL of the login form, but with the Antibot module this was blocked. And impossible to disable the module for testing as with Behat, hence the fact of making a connection by going to the login page and then filling in the form.

Integration into Gitlab CI

For the moment the tests are executable locally but not in Gitlab CI.

The problem is that while it would be possible to have a Cypress image as a service, the container that Gitlab CI runs commands on is the PHP container. And again, it's not clear that the problems for Docker Compose don't also arise in Gitlab CI.

Or unlike the PHPUnit "Functional JS" tests where PHPUnit is able to drive a Chromedriver container from the PHP container, this is not the case with Cypress. The PHP container cannot drive the Cypress container.

When I looked for examples of Cypress usage in Gitlab CI, I systematically came across examples where the application being tested was a JS application and therefore Cypress was directly on the NodeJS container where the application was installed.

I would therefore have to add NodeJS, Cypress, its dependencies and browsers to my CI PHP image, which would make it much heavier and for the use I have of it it's not worth it.

So I left this point aside.

Email tests

Similar to the Gitlab CI integration, when I was looking for information on email testing, every time it was a JS application running on the same server as Cypress.

So I left that out, but maybe with the Cypress Maildev plugin it would become testable.

To see if it's possible to easily add the plugin to the container without having to maintain a new image.

Comparison with PHPUnit and Behat

Positive point : debugging. Being able to take screenshots and have a video recording during test execution is a real plus compared to Behat. Less so compared to PHPUnit where PHPUnit exports each html page requested and allows debugging like any PHP code.

Cypress modifies the test execution environment where PHPUnit builds an environment for each test and the Behat extension for Drupal cleans up some modifications with methods including the "@AfterScenario" annotation.

As Cypress is used without a Drupal-specific extension, it is impossible to perform any actions other than those permitted by the site and its backoffice. Where with PHPUnit we have full access to the API and partial access with Behat (full if specific contexts are implemented).

Conclusion

I found Cypress very suitable for testing on the final application. It allowed me to discover a tool outside the PHP universe and independent of Drupal.

Cypress complements testing tools like PHPUnit and Behat. Each tool has its advantages and disadvantages.

Bonus

Slides from my presentation at the meetup in November 2021, finally the meetup took place before this article was written.