PHP Queues with php-resque

2nd May 2018 John Comments 1 comment

If you’ve been doing PHP development for more than a week you’ll have heard about the benefits of message queues. If you’re totally new to this world and have not heard of queues you should google for more info. At its most basic though a message queue allows your application to offload long running tasks to be completed as background tasks, thus speeding up the front end for your visitors.

For instance, on cutestix.co.za (a children’s photo competition site) when a user registers their little bundle of joy into the competition, there are 4 things which happen:

Entry is written to the database.
Image uploaded.
Image auto scaled and resized, etc.
Thank you email sent.

We could do all of that while the user is waiting (and indeed in version 1 of the site that’s what was happening). A better solution is to write the entry to the database and upload the image, then push the scaling operation and emailing to a queue and return to the user.

Resque Messaging Queue

We love the php-resque messaging queue because its easy to use and its based on Redis. Redis is an in memory data store. Because its in memory its blazingly fast!

In order to use Resque we do need to install a few things on the server

Installations:

You need to install Redis.

You can get installation instructions from https://redis.io/download. Much better, step by step instructions, including how to set Redis to start with your OS can be found here: How to Install and Configure Redis on Ubuntu

On our ubuntu system the installation was as simple as:

wget http://download.redis.io/redis-stable.tar.gz
tar xzf redis-stable.tar.gz
cd redis-stable
make
make test
sudo make install

2. Install PHPRedis.

PHPRedis is a c library extension for PHP to communicate with the redis server. Install it via PECL

pecl install redis

3. Install php-resque

PHP-resque is a PHP port of resque queue manager. Its a library file which can be installed via composer but for this article we’ll just clone it into our working directory.

git clone git://github.com/chrisboulton/php-resque.git

Once done I copied the contents of demo directory to the main php-resque directory and changed the path in resque.php to point to the correct resque directory.

Sending Emails with PHP-Resque

For the purposes of this example we’re going to put up two really simple forms which asks a user for their email address.

The first form, form-no-queue.html will submit to a php file, form-no-queue.php which sends the email while the user waits for the reply. The second form, form-queue.html submits the email job to php-resque and free’s up the user’s time.

Incidentally, source code for this little example can be found here: “PHP Rescue Example Code”

Changing our code to use the queue

Our form-no-queue.php code simply sends the email straight away like this (yes, I know there’s no validation, etc, but that’s a subject for another post):

include(dirname(__FILE__)."/classes/class.mailer.php");
$oMailer = new mailer();

$start = microtime(true);
$x = $oMailer->sendMail("This is a test message", "test mail", $_POST["email"]);

if( $x === true )
{
        print "Mail sending passed";
}
else
{
        print "Mail sending failed";
}

$end = microtime(true);

print "<p>Time elapsed in seconds: ".($end - $start);

Running this on an nginx server set up on my dinky laptop on a slow internet connection shows me a total run time of just over 4 seconds. That’s a pretty long time to make a user wait (I know, the world really does need some patience, but you’d need to be patient for that to happen!).

We now change our code to submit the emailing job to a resque message queue instead:

include(dirname(__FILE__)."/php-resque/lib/Resque.php");


$start = microtime(true);

$args = array("email" => $_POST["email"];

$id = Resque::enqueue("user:email", "sendEmail", $args);

if( $id === false )
{
        print "adding to the queue failed<p>";
}
else
{
        print "Job ID: ".$id."<p>";
}

$end = microtime(true);
print "<p>Time elapsed in seconds: ".($end - $start);

The first change is at line 1 where we include the Resque class. We then put the email address into an associative array we call $args.

We then push the job to the resque queue using the enqueue method. It takes 3 arguments:

Queue name (user:email)
The name of the class the with the code to run this queue (sendEmail)
Any args in an array, in this case the email address. This will be available in our sendEmail class as $this->args[“email”]

We now enter our email address on the enqueued form and hit submit. We instantly get back the result and our timer confirms a time of 0.0013480186462402 seconds (or 1.3 microseconds). Yay, we’ve just made our front end 4000 times quicker for our user but…

There’s no email!

The problem is that while we’ve added our job to the php-rescue job queue, there are no worker threads to execute the jobs.

Creating workers

To start a worker thread we need to be on the command line and navigate to the php-rescue directory.

We then call PHP cli with environmental variables which it will pass to php-resque like this:

cd /var/www/html/vendor/chrisboulton/php-resque/
sudo VVERBOSE=1 QUEUE=* COUNT=1 APP_INCLUDE="../../../queues/index.php" -u www-data php resque.php

Arguments:

VVERBOSE=1 – Turn on very very verbose mode so we can see things happening

QUEUE=* – Run all queues

APP_INCLUDE=”../../../queues/index.php” – Our autoloader file which calles our class. Remember, when we enqueued the job we enqueued it with a class of sendEmail.

We then invoke PHP as user www-data and the resque.php file which implements PHP Resque.

If all went well we should see some output on the console like this:

*** Starting worker ubuntu:1970:*
** [05:19:13 2018-04-30] Checking user:email
** [05:19:13 2018-04-30] Sleeping for 5
** [05:19:18 2018-04-30] Checking user:email
** [05:19:18 2018-04-30] Sleeping for 5
** [05:19:23 2018-04-30] Checking user:email

Great, our worker is now running and as you can see, its checking for jobs for our user:email queue.

If we now enter our email address on our php-queue.php for we should see our job being activated and sending our email:

** [06:28:32 2018-04-30] Checking user:email
** [06:28:32 2018-04-30] Sleeping for 5
** [06:28:37 2018-04-30] Checking user:email
** [06:28:37 2018-04-30] Found job on user:email
** [06:28:37 2018-04-30] got (Job{user:email} | ID: 0fa9351fc0bcd10a12396eaa75644021 | sendEmail | [{"email":"example@softsmart.co.za"}])
** [06:28:37 2018-04-30] Forked 2573 at 2018-04-30 06:28:37
** [06:28:37 2018-04-30] Processing user:email since 2018-04-30 06:28:37
** [06:28:41 2018-04-30] done (Job{user:email} | ID: 0fa9351fc0bcd10a12396eaa75644021 | sendEmail | [{"email":"example@softsmart.co.za"}])
** [06:28:41 2018-04-30] Checking user:email
** [06:28:41 2018-04-30] Sleeping for 5

Yes, our job was found. It called our sendEmail class and the job was done a few seconds later. It then just loops around and starts checking for new jobs.

And, checking my inbox the email did indeed arrive!

Setting up our class and autoloader

We have to take a slight step back here. One of the arguments when starting our php resque worker on the cli was: APP_INCLUDE=”../../../queues/index.php”. Also, when we enqueued our email in our PHP form, the second argument to the enqueue function was sendMail, our class name.

The way it works is that our autoloader included with APP_INCLUDE includes our class. Our class must implement a perform method and this is the method that the worker implements. As mentioned previously, the $args array is available as $this->args[“email”] in this class.

Here is what our autoloader looks like:

<?php
include(dirname(__FILE__)."/sendEmail.php");

and this is our class file:

<?php

class sendEmail
{

     function perform()
     {
          include_once("/var/www/html/classes/class.mailer.php");
          $oMailer = new mailer();

          $email = $this->args["email"];
          return $oMailer->sendMail("This is a test message sent from our queue", "test mail from queue", $email);

     }
     
}

So, as you can see, when we enqueue our job we give it the class name, in this case sendEmail. That class loaded by our autoloader (index.php) and the class implements a mandatory perform function. The perform function contains your custom coding. In this case it loads class and sends and email.

Daemonizing the worker threads

While its great that we managed to get our job queued and workers working, its not so convenient that we have to do that on the cli. What we want is a way to automate that.

There are various methods and you should sure take a look at the Fresque library but a simple method would be to put a start up job into the OSs start cron.

Monitoring your jobs with a GUI

You can install a web UI to view and monitor your jobs with the following commands:

apt-get install rubygems
gem install resque

This will install resque and you can view its web ui by starting resque-web from the cli by typing resque-web. This starts a server on port 5678.

SoftSmart

PHP, Laravel and Vue/Nuxt Developer