Cron Job Monitoring

Hello friends, in this content, I will talk about how we monitor our Cron Jobs. I have divided the article into three parts, so let’s get started if you’re ready.

Sections 📒

📌 What Was Our Goal?

📌 What Did We Use and Why?

📌 How Did We Do It?

What Was Our Goal?

We needed a monitoring system to assess the performance of our Cron Jobs. Our aim was to enable users to select active Cron Jobs from the user interface of our backend project and track their execution times and error statuses. Additionally, we wanted the ability to manually initiate these jobs using this functionality.

What Did We Use and Why?

I opted to create an MVC (Model-View-Controller) structure in the Kotlin project. To develop a dynamic website, I decided to use Thymeleaf on the Kotlin side.

Thymeleaf is a powerful template engine that works seamlessly with languages such as HTML, XML, JavaScript, and CSS. With this feature, it becomes easy to integrate data created on the server side into the user interface, enabling the rapid creation of dynamic web pages. Additionally, I chose to use Bootstrap on Thymeleaf; this allowed me to quickly build the user interface without the need for extra effort, resulting in a powerful and stylish design.

Among the core objectives of the project were the ability to select, add, and manually trigger cron jobs, as well as access relevant logs. During the listing and searching of logs, we utilized the features of jQuery DataTables. This way, users can easily access detailed information about cron jobs and effectively list and search log data.

The primary motivation behind choosing Couchbase for storing logs was its flexible data structure. This flexibility provides the option to move logs to more specific levels in later stages. Additionally, Couchbase optimizes job logs for real-time analysis, allowing fast access to specific data. Advanced querying capabilities enable us to extract meaningful information from the logs, providing a significant advantage for our project. Currently, we have decided to store a maximum of 100 logs for a cron job, as we believe older logs will no longer be useful.

How Did We Do It?

Firstly, I wrote a controller, and in the initial stage, I needed to list all my jobs.

The endpoint takes a model because it facilitates the transfer of data created by the controller to the view layer. The Model class contains the necessary data to be merged with a template. Then, using the applicationContext that I injected, all beans of the type BaseJob in the class are collected and assigned to beansOfType. By sending beansOfType.keys to my model, I ensure that all my cron jobs are listed. I will return jobs because I will create jobs.html under src → main → kotlin → resource→templates. As an example, my output will look like this. You can use this template if you prefer, but as I mentioned, I used Bootstrap and listed all the crons in a popup, making it possible to click and trigger that cron. Below is a simple listing example:

Sure, let’s move on to the create stage. I created a class of type JobRequestForm.

Then, from the UI, I need to send the Cron name to my endpoint.

Here, the endpoint will receive the model name from the UI, and after creation, it will redirect back to the jobs page. On the UI side, you can do something like the following:

Certainly, we can write an endpoint for the scenario where an admin selects a job, posts the form, and then needs to manually trigger the created job. Here is an example of such an endpoint:

@ModelAttribute(“jobRequest”) annotation ensures that the data coming from the form is bound to the model property named jobRequestForm. Using applicationContext.getBean(jobRequestForm.name!!) as BaseJob, we retrieve the relevant Spring bean by using the job name from the form (jobRequestForm.name). Assuming that this bean is of type BaseJob (as BaseJob), it is assigned to the variable selectedJob.

job.execute(): By calling the execute() method on the obtained bean of type BaseJob, it ensures the execution of the corresponding task.

While executing jobs, it is necessary to log messages. So, how can we achieve this? We need to write and send the necessary messages during the execute stage of the jobs.

That sounds like a good practice. Sending the necessary messages to your log method during job execution and recording info or error statuses helps in tracking and monitoring the job processes effectively. If you have any specific questions or if there’s anything else you’d like assistance with regarding this process or any other topic, feel free to let me know!

This way, we can easily log specific stages during the logging processes. Particularly, key information crucial for us, such as job ID, time, and status, has been identified in these logs. Our next step is to list our cron-specific logs. Just like the UI examples above, when clicking on any job, the name of the job falls into the endpoint below.

By retrieving job log information, I can transmit these details to the user interface (UI). To process this information on the UI side, I will list the status, time, and message fields. This way, on the user interface, I can display job logs in a detailed manner.

With this structure, I was able to achieve advanced functionality on the user interface using DataTables, such as searching and sorting based on time. Of course, don’t forget to add the CDN for this. Below, you can take a look at an example DataTable:

The screen on the right lists all of my jobs, and I can access or trigger the data for the desired jobs as shown in the screenshot above. I can also search through the logs based on time, status, and message. This way, there will no longer be overlooked jobs in case of an error, and monitoring becomes more straightforward.

I would like to extend special thanks to my teammate Çağatay Gökçel, whose contributions and encouragement led me to explore and improve job monitoring. I also appreciate the reviews and contributions from Enes Açıkoğlu, Onur Ay, and Yakup Emre Tanrıkulu. 🧡

Until the next article, stay well! 😊