Laravel allows you to be able to make all kinds of systems from simple to complex, many times these tasks involve a high computation, and it is one of the main disadvantages that we have in web applications with respect to desktop applications, and that is that applications web, usually (the traditional ones at least) have fewer computing resources available, or are more limited and this is something that we see in doing simple operations such as sending an email or processing an image, than when using the main server thread (the same that we use when making requests to the server) we see that the server takes a few seconds to return a response and it is this additional time that is used to complete these tasks. Additionally, if the operation to be performed exceeds the computing capacity or many people perform operations of this type at the same time, the server may give a time exhaustion error.
From what was mentioned in the previous paragraph, we must use a mechanism to be able to do all these processes efficiently, and the way we do this is by delegating these operations, instead of making them the main thread, we can. Send these tasks to a process (or several depending on how we configure it) which is responsible for processing tasks consecutively one after the other and with this, we already introduce the definition of queues and jobs.
Jobs are these expensive operations at the computational level, sending an email, processing an image, processing an Excel file, generating a PDF, etc, which are assigned or sent to a queue or queues (depending on how we configure it) and they process efficiently, that is, we can enable a certain number of secondary processes that are managed by the queues to process these tasks. Therefore, with this system, it does not matter how many jobs exist at the same time, they will be processed from time to time xxx little without saturating the system. Additionally, it is possible to specify job priorities so that they are executed before others.
In short, by using the system of queues and workers, the response capacity is improved, it is possible to horizontally increase the number of workers available to process these jobs, it is possible to re-execute failed jobs, thereby giving better tolerance to system failures, all this asynchronously without affecting the main thread, so with the importance of this system clarified, let's learn how to implement it.
Queue controller
First, we must choose a strain controller to use among all the existing ones:
- 'sync': The 'sync' handler executes the queued jobs right after and in the same request-response cycle. It is suitable for development and testing environments, but not for production.
- 'database: The 'database' driver stores queued jobs in a database table in a separate queue worker process.
- 'redis' - The 'redis' driver uses Redis as a queue store.
- 'beanstalkd': The 'beanstalkd' driver uses the Beanstalkd queue service to process queues.
- 'sqs' (Amazon Simple Queue Service): The 'sqs' driver is used for integration with Amazon SQS.
- 'rabbitmq' - The 'rabbitmq' driver allows RabbitMQ to be used as a queue driver.
- 'null' - The null handler is used to disable the queuing system completely.
By default, the database is configured:
config\queue.php
'default' => env('QUEUE_CONNECTION', 'database')And to carry out the following tests, you can use the database although, usually Redis is an excellent option because it is a fast and efficient database that we installed previously and is the one we will use:
config\queue.php
'default' => env('QUEUE_CONNECTION', 'redis')Finally, we start the process to execute the jobs by:
$ php artisan queue:workAnd we will see through the console:
INFO Processing jobs from the [default] queue. Every time a job is processed, you will see in the terminal:
.................................................................... 3s DONE
2024-07-12 09:44:31 App\Jobs\TestJob ............................................................................... RUNNING
2024-07-12 09:44:34 App\Jobs\TestJob ............................................................................... 3s DONE
2024-07-12 09:45:43 App\Jobs\TestJob ............................................................................... RUNNINGIt doesn't matter if you dispatch jobs from your controllers or similar and the queue is NOT active, Laravel registers them and when you activate the queue process, it dispatches them; and this is all, with this Laravel sets up a process to manage the jobs, we need to create the job that we will see in the next section.
In this post we are going to learn about the use of Queues and Jobs in Laravel (Queues and Jobs) to be able to perform jobs in the background.
The idea of this API is that we can postpone tasks (leave them in the background) that require a high computational consumption and all this is to improve the user experience and avoid application crashes; in such a way that these high computing tasks will be executed in another thread or process in our server and will be resolved as they arrive or depending on their priority.
Basically all this process is carried out between the tasks/jobs (the high computational processes) and the queues (mechanism to register the jobs).
In the same way, we are going to talk a little more in detail about these two fundamental components.
The jobs and the queues
Jobs are the heavy or processing-intensive task that we want to defer; and they can be of any type such as massive or simple sending of emails, processing images, videos, etc; therefore these are tasks that we register as pending so that Laravel processes them one by one as it becomes available; so the next thing you can ask yourself is, how is this organization managed?; that is, who is in charge of managing the priority between tasks or jobs (queues) and how do we maintain an organization of all this.
The jobs
In Laravel, jobs are an important part of the framework's job queue functionality, that is, jobs are used to process the heavy jobs stored by “queues”; these heavy tasks can be anything like sending emails, processing images, videos, among others; this is ideal since these tasks are not part of the user's main session and it avoids consuming resources from the user's main session, apart from avoiding the famous "not responding" windows and in general, you have a better experience in using the app as it does not affect the performance or responsiveness of the main app.
Jobs in Laravel can be created to perform whatever task is needed in the back-end without the user having to wait for the task to complete; that is, how to send an email or export data in an excel or a similar format.
The great thing about jobs is that they are executed sequentially, suppose you create a job to process emails, the emails are sent one by one without the need to overload the page, which would not happen if the email delivery was handled from the user's main session and suddenly, 5 or more users come to send emails from the application, therefore, at the same moment of time, you would have multiple emails and spend the resources that this requires.
In short, jobs in Laravel are an essential part of the framework's job queue, allowing heavy or non-essential tasks to be performed in the background to improve the performance and responsiveness of the main application.
The queues
Already at this point, we have introduced the concept of "queues" in the use of jobs, in the same way, we are going to explain it quickly; the job queues correspond to the operation that you want to perform, which, as we mentioned before, corresponds to the "heavy" operation that you want to perform, that is, instead of sending an email from a controller, it is stored in a queue that is then It is processed by another process.
Create your own queues in Laravel to maintain order
The answer to what was commented above is quite simple, it would be the queues as a mechanism that allows registering or adding these works or jobs; we can register or have as many queues as we want and we can specify to Laravel via artisan which queues we want to process first; for example, let's look at the following example:
php artisan queue:work --queue=high,defaultThrough the previous command we are telling Laravel to first process the jobs from a queue called "hight" and then from a queue called "default", which by the way, is the queue that we have defined by default.
Connections for the structure to maintain the tails
The connections are the default way we have to indicate how the entire process of the queues is going to be carried out (since it remembers that the queue absorbs the task, therefore, at this point, once the task or job is registered, in a queue, in our queue the task no longer paints anything); but to be able to use all this structure we must indicate the connection and with it we must indicate:
- The controller or driver
- Default or Configuration Values
Therefore, through the connections we can specify the driver that we are going to use, which can be any such as the database or other services; for example:
Database:
- Amazon SQS: aws/aws-sdk-php ~3.0
- Beanstalkd: pda/pheanstalk ~4.0
- Redis: predis/predis ~1.0 or phpredis PHP extension
And some extra parameters for configuration.
Configure the Driver or connector of the queues for the database
Now we are going to open the config/queue.php file and the .env, in which we are going to look for a configuration called QUEUE_CONNECTION
Which by default is called QUEUE_CONNECTION and has the sync configuration, which if you analyze it a bit you will see that we have several types of connections that we can use, including the database call that is the one we are going to use, therefore configure it at least in the .env file; in my case I'm going to leave it like this in the .env:
QUEUE_CONNECTION=database
Y en el config/queue.php quede:
'default' => env('QUEUE_CONNECTION', 'database'),Create the jobs table in our database
Now the next thing you can ask would be, how can we configure the table in our database?; following the official documentation:
php artisan queue:table php artisan migrateWe already have an artisan command that helps us in this whole process, which would be the one that allows us to generate the migration to create the table; and with this we are ready.
Creating our first Job in Laravel
Now we are going to create our first Job that we are going to configure with the default queue; for that, we are going to create one using our artisan:
php artisan make:job ProcessImageSmallAnd with this we have a file generated in App\Jobs
Which consists of two main blocks:
- __construct
- handle
The constructor function to initialize the basic data that we are going to use and the handle function to do the heavy processing; It's that simple, therefore in the handle function we can send emails, process videos or images, for our example we are going to assume that we have an image that we are receiving by parameters:
protected $image;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct(PostImage $image)
{
$this->image = $image;
}This image is basically an instance of a model that looks like this:
class PostImage extends Model
{
protected $fillable = ['post_id', 'image'];
public function post(){
return $this->belongsTo(Post::class);
}
public function getImageUrl(){
return URL::asset('images/'.$this->image);
//return Storage::url($this->image);
}
}And now, we are going to do a heavy process like scaling an image; for this, we are going to use the following package that we can install using composer:
composer require intervention/imageThe code that we are going to use is basically self-explanatory:
<?php
namespace App\Jobs;
use App\PostImage;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Intervention\Image\ImageManagerStatic;
class ProcessImageSmall implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $image;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct(PostImage $image)
{
$this->image = $image;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
$image = $this->image;
$fullPath = public_path('images' . DIRECTORY_SEPARATOR . $image->image);
$fullPathSmall = public_path('images' . DIRECTORY_SEPARATOR . 'small' . DIRECTORY_SEPARATOR . $image->image);
$img = ImageManagerStatic::make($fullPath)->resize(300,200);
$img->save($fullPathSmall);
}
}
In which we simply create an instance of our package, which we installed earlier, specify the location of the image and using the resize function scale the image to the specified dimensions; then we save the new image and with this we are ready.
Dispatch a job
The next thing we have to specify is where we are going to call this job, which can be anywhere in our application, such as when we load an image, etc; suppose we have a method to load an image in Laravel for example:
public function image(Request $request, Post $post)
{
$request->validate([
'image' => 'required|mimes:jpeg,bmp,png|max:10240', //10Mb
]);
$filename = time() . "." . $request->image->extension();
$request->image->move(public_path('images'), $filename);
//$path = $request->image->store('public/images');
$image = PostImage::create(['image' => /*$path*/ $filename, 'post_id' => $post->id]);
ProcessImageSmall::dispatch($image);
return back()->with('status', 'Imagen cargada con exito');
}From here; If you look closely at the code, you'll see that we define an instance of our queue with the dispatch method to dispatch the job ProcessImageSmall::dispatch($image)
Raise daemon to process queues
Now if we review the previous table, we will see that a job was registered, but it has not been dispatched, to dispatch it, we have to activate the daemon that is in charge of doing this job, start the process that is in charge of doing the jobs pending in the jobs table:
php artisan queue:workAnd with this we will see that we have everything ready and the job has already been dispatched:
With this, we learned how to use jobs and queues in Laravel; remember that this is one of the many topics that we deal with in depth in our Laravel course that you can take on this platform in the courses section.
- Andrés Cruz
En español
Develop with Laravel, Django, Flask, CodeIgniter, HTML5, CSS3, MySQL, JavaScript, Vue, Android, iOS, Flutter