A lot has been said about the importance of “off-lining jobs,” which is essentially deferring heavy, time and resource consuming transactions to the background. When using offline processing, users don’t have to wait for a transaction to complete while accessing your website or application. We see this commonly in credit card transactions, email creation, and PDF generation. Additionally, offline processing helps you with the ability to better control the load and utilization of resources at peak load times.
Off-lining activities usually means a background system for managing queues is required as part of the setup. It can be based on the basic operating system scheduler (like cron, a time-based job scheduler), self-developed systems, or commercial ones (like BMC’s Control-M or Zend Server Job Queue).
Job creation and system configuration
Taking selected parts of the code for asynchronous execution as an offline job is a worthwhile, reasonable effort conducted by technical code changes. Once the desired parts are defined as jobs, it is up to the job system to handle the queuing and execution. Job systems can usually be configured to manage several queues, several job servers, and priorities.
Zend Server Job Queue can be configured to handle jobs and queues priorities, scheduling and reoccurrence, execution concurrency, managing retries and failures, and more.
Let’s assume you’ve chosen your system and created the jobs as suggested. And now you ask yourself:
- Are you done with it?
- How do I configure my Job Queue system?
- Is this optimized for my needs?
- How many jobs failed and why?
- How long are the jobs waiting before they can run?
Tough questions, hard to answer. Most systems come with default settings, but usually the system basic configuration is not good enough – no one size fits all.
Do you have to master queueing theory to optimize your job queue system and get results? Definitely not.
Optimize offline jobs using Zend Server Job Queue Pulse
Zend Server 9.1 introduces Job Queue Pulse, which collects the analytical information and insights obtained by monitoring the entire Job Queue system and visually presents them.
Job Queue Pulse displays high-level statistics such as the number of jobs that ran, average number of jobs per minute, and average wait time per job. This information provides clarity on how loaded your job system is. In cases of high wait times, you can increase the number of queues or the concurrency, thus increasing the number of jobs that run in parallel. On the other hand, if the wait times are very low, you may consider reducing the concurrency level, to reduce the load on the backend systems (e.g. email server).
Job Queue Pulse displays the success rate and detailed information about failures. Clicking the graphs drills down to a detailed list of jobs, so you can continue analyzing which jobs failed and why. In that context, Pulse is integrated with the Zend Server monitoring system, where it collects the most critical events that are related to the jobs, so you can decide what to handle and solve first. Obviously, you can filter the view per period, application, or queue.
To conclude, by using Job Queue Pulse, you can better optimize, analyze, and fine-tune asynchronous and scheduled Jobs processing. The result is better resources utilization and improved end-user customer experience.