Update
: Node.js workers are now available in LTS versions and there’s a new package Bree that makes use of it! It makes job scheduling a breeze. I’ve updated the blog post that covers job scheduling with Bree using Workers. Alternatively, there’s also an implementation with Kue with which this post was based. I would not recommend it, since it’s no longer maintained.
Bree
Bree comes with so many features right out the box. It’s a job scheduler for Node.js with support for cron, dates, ms, later, and human-friendly strings. Bree uses workers and spawns sandboxed processes, supports async/await, retries, throttling, concurrency, and cancelable jobs (graceful shutdown)! It’s well suited for production environments at scale.
Why Bree?
The foreword on their GitHub repository explains it beautifully,
“Before creating Bree, I was a core maintainer (and financially invested in development) of Agenda . I have been with the Node.js community for a very, very long time, and have tried literally every solution out there (see Alternatives that are not production-ready ). I have found that all existing solutions are subpar, as I have filed countless issues; discovered memory leaks found functionality not working as described, unresolved core bugs have persisted over time, etc.
Previous to creating this, I was relying heavily on bull ; having created @ladjs/bull – but due to core issues (and being Redis-backed) it was not the best tool for the job. Bull might have been okay if the core issues were fixed, however since it uses Redis it should not be used for a job queue. From my experience, Redis should only be used for caching and session storage purposes (e.g. CDN or managing user login state in your application).
Since workers are now readily available in LTS versions of Node, I thought it would be a great time to implement them in a job scheduler environment. Additionally, my research and development of a better anti-spam and anti-phishing classifier with Spam Scanner gave me some necessary insight to using workers .
Bree was created to give you fine-grained control with simplicity and has built-in support for workers, sandboxed processes, graceful reloading, cron jobs, dates, human-friendly time representations, and much more. We recommend you to query a persistent database in your jobs, to prevent specific operations from running more than once. Bree does not force you to use an additional database layer of Redis or MongoDB to manage the job state. In doing so, you should manage the boolean job states yourself using queries. For instance, if you have to send a welcome email to users, only send a welcome email to users that do not have a Date value set yet for welcome_email_sent_at
.”
Bree was created by @niftylettuce , a prolific GitHub and npm contributor who is also an Express Technical Committee Member and Core Member.
Setting up the project
This example project will send an email after a person books a movie ticket. The first email will be the confirmation of the booking which will be sent out as soon as the movie is booked. The second email will be a reminder email that will be sent 10 minutes before the movie timing by a job scheduled using Bree.
This example simply writes to the file system. To use as a persistent database, however you could plug in MongoDB (e.g. Mongoose), SQL, Redis, or any other storage mechanism you desire.
We’ll be using npm init
to set up a Node JS project.
First things first, let’s create a directory for your project. I have assumed the project name as node-js-job-scheduling
. Using mkdir node-js-job-scheduling
, create your directory.
Run npm init
until the process is finished. You can also add required attributes if you like to. You had nothing, but you have this now.
Let’s call NPM for help
We’ll be installing some npm packages to set up a simple express server and test out an API endpoint. Let’s hope it responds to our call on the first try.
Install express & body-parser to set up a simple server that listens to a port.
Just Listen
Create a index.js
file in the root of the project and set up a simple application like the following:
Now, curl http://localhost:8080/test
will result with a success response. All good. For more information on the setup, please refer index.js from the example repository.
Email-templates for Node.js emails
Previously, I’d have used Nodemailer for emails, but Email-Templates are a lot more easier and it does not use any third parties which makes it easier for development.
Now, we’ll write a test endpoint that sends out an email.
Now, curl -X POST http://localhost:8080/send-email -d "[email protected]"
will respond with a success response, and the email preview will be opened in your browser.
Bree for Job Scheduling
Usage
The example below assumes that you have a directory called “jobs” at the root of the directory. Inside the jobs directory are individual scripts that are run using workers.
Let’s create a file called bree.js
. Make sure you run node bree.js
.
name
: The name of the job. This should match the base file path (e.g. foo if foo.js is located at /path/to/jobs/foo.js) unless path option is specified. A value of index, index.js, and index.mjs are reserved values and cannot be used here.interval
: Default interval for jobs (e.g. a value of 0 means that there is no interval, and a value greater than zero indicates a default interval will be set with this value). This value does not apply to jobs with a property of cron.
For more information, check out the documentation .
Now, we’ll have /jobs/email.js
that sends out a reminder email.
Schedule a Node JS Job
We’ll now write an API that accepts the user email and the time of the movie (usually fetched from DB) and add a job to the queue to send out the reminder Email. Inside index.js
,
Now, a job will be scheduled which will send a reminder email 10 minutes before the movie time
.
To test the endpoint, we can use curl -X POST http://localhost:8080/book-ticket -d "[email protected]" -d "start_time=7/14/20 4:00 PM"
that’ll send out an email when the ticket is booked, and queue an email that needs to be sent 10 minutes before the movie time.
I personally enjoyed how much Bree makes my job to schedule a job easier. You can find the Source Code in this GitHub Repository . I’m on Twitter , Say Hi to me to talk about programming languages, compilers, Rust & a lot more. 😃
You can do a lot more with Bree. Here’s an example from their documentation.
For more information, check out their documentation .
Alternative: Kue [No Longer Maintained] & Redis
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, geospatial indexes with radius queries and streams.
To be precise, it is an in-memory key-value database.
We’ll be using Redis along with a Node JS package called Kue to schedule background process for a simple node js application. Background process usually helps in sending an Email/ notification or running any background process. For instance, in a movie ticket booking application, you need to notify a customer via an email when a seat is booked and remind him/her 10 minutes before the movie time. This will be our use case.
In this article, we’ll write a minimal REST API server with express js. The prerequisite to this blog will be to know how Node JS works and make sure that Redis is installed and running.
Setting up the project
We’ll be using npm init
to set up a Node JS project.
First things first, let’s create a directory for your project. I have assumed the project name as node-js-job-scheduling
. Using mkdir node-js-job-scheduling
, create your directory.
Run npm init
until the process is finished. You can also add the required attributes if you like to. You had nothing, but you have this now.
Let’s call NPM for help
We’ll be installing some npm packages to set up a simple express server and test out an API endpoint. Let’s hope it responds to our call on the first try.
Install express & body-parser to set up a simple server that listens to a port.
npm install express body-parser —save
Just Listen
Create a index.js
file in the root of the project and set up a simple application like this:
Let’s use Postman to test our API endpoints. To start our server, use nodemon
or a command like node index.js
. It’s recommended to use nodemon as that’ll automatically restart the server after every change made to the code.
Let’s check if the endpoint responds to our call, open Postman try this endpoint - http://localhost:8080/test
Nodemailer for Mails
Let’s install nodemailer to our project with the help of NPM.
npm i nodemailer —save
Create a file named nodemailer.js
in the root of the project and add the following code that’ll send an email to the recipient when invoked. Let’s assume that this method only takes one argument which is the recipient’s email.
Let’s test out if the Mail part works. We have used an ethereal
username and password which will not send out an actual email but yet mimics it. We’ll be getting a link on how the mail will look.
Create an endpoint that invokes the method to send out an email. Before that let’s require our method in index.js
const nodemailer:require(’./nodemailer’)
Test our endpoint in postman using POST http://localhost:8080/send-email
with the following body,
The response will be the E-mail preview link from ethereal
-
Kue for Job Scheduling
Let’s install the required NPM package to schedule a background process. For that, we need to install kue
which is a priority job queue backed by Redis, built for node.js.
We’ll be writing an endpoint to schedule the background process/job and a worker that’ll process it. For our requirement, it will be sending an email. Let’s assume that a user is booking a ticket for a movie. He/she should be notified twice i.e. When the booking is confirmed (at the time of booking) and 10 minutes before booking.
For refactoring, we’ll separate the job schedule as two different methods. One will be to schedule the job and the other will be to process the scheduled job.
We’ll create a kue.js
file that contains the following code block,
The above code will schedule a job which will assign a name and it’s params. Also the delay. We will have one-second delay for rolling out the booking confirmation email and milliseconds timestamp which will be 10 minutes before the start time of the movie.
Job Workers
We’ll create another file named worker.js
which will process the scheduled job based on its names. So, in our case, it will be the email job which will roll out emails.
The above code block will make use of the nodemailer utility function we defined previously in this article and send out the email. Note that the arguments/ parameters we passed when scheduling the job will be as an attribute in the first argument of the queue process callback function with the key data
. So, we get that with the help of object destructuring - let { data }:job;
, and pass it to the nodemailer utility function. Require/Import the worker in index.js
file which is our project’s entrypoint.
API Endpoint
We’ll now write an API endpoint /book-ticket
that will schedule the job for us in index.js
. It’ll schedule two jobs for us. One is to send out the booking confirmation E-mail that has a delay of say, one-second and the second one is to send an email 10 minutes before the booking.
The above endpoint makes use of the utility function we wrote to schedule a job. It has a job name, the time (delay) and the email args which is required to send out the email.
The first job is about the confirmation email after a ticket is booked with a delay of one second which is 1000 milliseconds. The second job is about the notification email that should be sent 10 minutes before the start time of the movie. So the delay time is (req.body.start_time - 10 * 60) * 1000
where the start time is a unix timestamp
from the request body. The request body will also contain the email of the user to whom it should be sent to.
The request body will be as the following,
Let’s test it again with Postman
The Ethereal preview link will be printed in the console as follows,
The confirmation E-mail
The Notification E-mail
Folder Structure
Our journey ends here and I hope you found this article helpful. You can also find the project on Github or follow me on Twitter for some funny retweets.