Asynchronous fault tolerant programming with PHP

In the previous post on fault tolerant programming in PHP we have shown the basics of fault tolerant programming using the circuit breaker pattern. Now we will show you a running demo of the application where the circuit breaker library Phystrix is combined with asynchronous programming. The advantage of this approach is that it allows the querying of multiple backend services asynchronously. A timeout can be set for each of the calls and the circuit breaker will deal with failing services.

In order to get this to work we had to make a few changes to Phystrix. The code for this demo can be found here, also check out a screencast of the demo below.

A quick review

Before we get started, lets recap the example from the previous post. It consists of two internal and one external API:

The problem we showed was that by just programming the happy path an error in one of your dependencies can propagate all the way through your application:

If the Inventory service app is down the Public shop API will also be down. We showed you how to avoid this using the circuit breaker pattern, making the application more fault-tolerant. The next step is to be able to use the circuit breaker pattern for asynchronous calls. In order to get asynchronous calls in php we will use the ReactPHP library. Our demo shows how everything works.

Running the demo

Clone the repository and use composer to install the dependencies:

$ git clone https://github.com/qandidate-labs/phystrix-async-demo.git
$ cd phystrix-async-demo
$ composer install

You can now run the demo - be sure to execute each command in its own terminal.

$ bin/run_all.sh
$ bin/get_catalogue_inventory_status.sh

Both of the commands will show some output. run_all.sh will output the server access logs and the Phystrix command log:

The command log shows that the execution of both the GetCatalogueCommand and the GetInventoryStatusCommand resulted in a successful call. get_catalogue_inventory_status.sh has the following output:

Since both microservices are fully operational the catalogue and inventory status are properly displayed. In order to see the circuit breaker in action we will make the inventory status API fail. You can do this by modifying inventory-status/index.php and by adding a sleep(1). The inventory status API will take too long to respond, triggering a timeout. When the API call fails, Phystrix will return fallback values instead of propagating the failures.The result is that the inventory status of all items in the catalogue is set to -1.

The really interesting part can be found in the command log:

First, a few successes are shown. After this, a failure is triggered for the GetInventoryStatusCommand (due to the timeout). This results in a FAILURE,FALLBACK_SUCCESS. After two failures the circuit is short circuited (SHORT_CIRCUITED,FALLBACK_SUCCESS) - the result is that the fallback is returned directly. After some time a FAILURE,FALLBACK_SUCCESS occurs again. The circuit breaker decides it is time to test if the service is back up. Since it is not (the timeout is still hit), the Command fails again, and the circuit remains short-circuited. Had the retry been a success, the circuit would have been closed and all new calls would have again be forwarded to the backend service.

You now have seen the demo and everything appears to be working magically, including the asynchronous backend calls. Next time we will have a look at how everything works internally!

TL;DR

This demo shows the concept of having a circuit breaker for asynchronous api calls. This prevents a failure in one of your calls to take down your whole system. This screencast shows the circuit breaker in action.