PowerJob brief introduction

Product characteristics

PowerJob It is a new generation of distributed task scheduling and computing framework , Its main functions are as follows :

• Easy to use ： Provide front end Web Interface , Allow developers to visually complete the management of scheduling tasks （ increase 、 Delete 、 Change 、 check ）、 Functions such as task running status monitoring and running log viewing .
• Perfect timing strategy ： Support CRON expression 、 Fixed frequency 、 Fixed delay and API Four timing scheduling strategies .
• Rich execution patterns ： Support single 、 radio broadcast 、Map、MapReduce Four execution modes , among Map/MapReduce The processor enables developers to obtain the ability of cluster distributed computing with only a few lines of code .
• Workflow support ： Support online configuration of task dependencies （DAG）, Arrange tasks visually , At the same time, it also supports data transmission between upstream and downstream tasks .
• Actuators support a wide range of applications ： Support Spring Bean、 built-in / External location Java class , In addition, the dependency package provided by the government can be introduced , One click Integration Shell、Python、HTTP、SQL Etc , Wide application .
• Convenient operation and maintenance ： Support Online log function , The log generated by the actuator can be displayed in real time on the front console page , Reduce debug cost , Greatly improve development efficiency .
• Rely on streamlining ： Relational databases are only minimal （MySQL/PostgreSQL/Oracle/MS SQLServer…）
• High availability & High performance ： The dispatching server is carefully designed , Change the strategy based on database lock in other scheduling frameworks , Realize lockless scheduling . Deploying multiple scheduling servers can achieve high availability and performance improvement at the same time （ Support unlimited horizontal expansion ）.
• Failover and recovery ： After a mission fails , Retry can be completed according to the configured retry policy , As long as the actuator cluster has enough computing nodes , The task can be successfully completed .

Online trial ：http://try.powerjob.tech/

apply name ：powerjob-agent-test

password ：123

Applicable scenario

• Business scenarios with regular execution requirements ： Such as full synchronization of data every morning 、 Generate business reports, etc .
• There are business scenarios that need to be executed by all machines ： For example, use broadcast execution mode to clean up cluster logs .
• There are business scenarios that require distributed processing ： For example, a large amount of data needs to be updated , Single machine execution takes a long time , have access to Map/MapReduce The processor completes the distribution of tasks , Mobilize the whole cluster to speed up computing .
• Yes Delay the Business scenarios for certain tasks ： Such as order expiration processing .

Design objectives

PowerJob The design objective of the project is Enterprise level distributed task scheduling platform , That is, become an internal Task scheduling Middleware . The whole company uniformly deploys the dispatching center powerjob-server, All its business line applications only need to rely on powerjob-worker You can access the dispatching center to obtain task scheduling and distributed computing power .

Comparison of products of the same type

	QuartZ	xxl-job	SchedulerX 2.0	PowerJob
Timing type	CRON	CRON	CRON、 Fixed frequency 、 Fixed delay 、OpenAPI	CRON、 Fixed frequency 、 Fixed delay 、OpenAPI
Task type	built-in Java	built-in Java、GLUE Java、Shell、Python Etc script	built-in Java、 External location Java（FatJar）、Shell、Python Etc script	built-in Java、 External location Java（ Containers ）、Shell、Python Etc script
Distributed tasks	nothing	Static fragmentation	MapReduce Dynamic segmentation	MapReduce Dynamic segmentation
Online task governance	I won't support it	Support	Support	Support
Log whitescreen	I won't support it	Support	I won't support it	Support
Scheduling mode and performance	Based on database lock , There are performance bottlenecks	Based on database lock , There are performance bottlenecks	An unknown	Unlocked design , Strong performance, unlimited
Alarm monitoring	nothing	mail	SMS	mail , Provide interfaces that allow developers to extend
The system depends on	Relational database （MySQL、Oracle…）	MySQL	RMB	arbitrarily Spring Data Jpa Supported relational databases （MySQL、Oracle…）
DAG workflow	I won't support it	I won't support it	Support	Support

* Basic concepts

Group concept

• appName： apply name , It is recommended to connect with the actual user PowerJob The application name of the is consistent , be used for Business grouping and isolation . One appName Equal to a business cluster , That is, the actual one Java project .

The core concept

• Mission （Job）： Describes the need to be PowerJob Scheduled task information , Include task name 、 Scheduling time 、 Processor information, etc .

• Task instance （ JobInstance, abbreviation Instance）： Mission （Job） After being scheduled for execution, a task instance will be generated （Instance）, The task instance records the runtime information of the task （ The relationship between task and task instance is similar to the relationship between class and object ）.

• Homework （Task）： The execution unit of the task instance , One JobInstance There is at least one Task, The specific rules are as follows ：

• • Stand alone tasks （STANDALONE）： One JobInstance Corresponding to one Task
  • Broadcast task （BROADCAST）： One JobInstance Corresponding N individual Task,N Number of cluster machines , That is, each machine will generate a Task
  • Map/MapReduce Mission ： One JobInstance Corresponding to several Task, Manually by the developer map produce

• workflow （Workflow）： from DAG（ Directed acyclic graph ） Describe a set of tasks （Job）, For task scheduling .

• Workflow instance （WorkflowInstance）： After the workflow is scheduled for execution, a workflow instance will be generated , Records the runtime information of the workflow .

Expand the concept

• JVM Containers ： With Maven The dimensions of engineering projects organize a pile of Java file （ Developers develop many Java processor ）, It can be dynamically published through the front-end web page and loaded by the actuator , It has strong expansion ability and flexibility .
• OpenAPI： Allow developers to complete manual operations through interfaces , Make the whole system more flexible . Developers can build on API Easily expand PowerJob The original function .

Scheduled task type

• API： This task will only be performed by powerjob-client Provided in OpenAPI Interface trigger ,server Will not actively schedule .
• CRON： The scheduling time of this task is determined by CRON Expression assignment .
• Fixed frequency ： Second level task , Run every few milliseconds , Function and java.util.concurrent.ScheduledExecutorService#scheduleAtFixedRate identical .
• Fixed delay ： Second level task , How many milliseconds is it delayed to run once , Function and java.util.concurrent.ScheduledExecutorService#scheduleWithFixedDelay identical .
• workflow ： This task will only be executed by the workflow schedule to which it belongs ,server The task will not be actively scheduled . If the task does not belong to any workflow , The task will not be scheduled .

remarks ： Fixed delay and fixed frequency tasks are collectively referred to as second level tasks , These two tasks cannot be stopped , Only when the task is closed or deleted can the task be stopped .

Project address

PowerJob Main item ：https://github.com/PowerJob/PowerJob

PowerJob The front-end project ：https://github.com/PowerJob/PowerJob-Console

PowerJob Official website project ：https://github.com/PowerJob/Official-Website

Project architecture description

This project consists of the main project （PowerJob） And front end projects （PowerJob-Console） constitute , among ,PowerJob Each module is described below ：

├── LICENSE
├── powerjob-client // powerjob-client, Ordinary Jar package , Provide  OpenAPI
├── powerjob-common //  Common dependencies of each component , Developers don't need to feel 
├── powerjob-server // powerjob-server, be based on SpringBoot Implementation of the scheduling server 
├── powerjob-worker // powerjob-worker,  Ordinary Jar package , Access powerjob-server Your application needs to rely on this Jar package 
├── powerjob-worker-agent // powerjob-agent, Executable Jar file , Direct access to powerjob-server Agent application for 
├── powerjob-worker-samples //  Tutorial project , It contains all kinds of Java Writing examples of processors 
├── powerjob-worker-spring-boot-starter // powerjob-worker  Of  spring-boot-starter ,spring boot  Applications can generally introduce this dependency on one click access  powerjob-server
├── powerjob-official-processors //  Official processor , Contains a series of commonly used  Processor, Rely on this  jar  The bag can be used 
├── others
└── pom.xml