EngineConn architecture design
EngineConn: Engine connector, a module that provides functions such as unified configuration management, context service, physical library, data source management, microservice management, and historical task query for other microservice modules.
EngineConn architecture diagram
Introduction to the second-level module:
linkis-computation-engineconn interactive engine connector
The ability to provide interactive computing tasks.
Core class | Core function |
---|---|
EngineConnTask | Defines the interactive computing tasks submitted to EngineConn |
ComputationExecutor | Defined interactive Executor, with interactive capabilities such as status query and task kill. |
TaskExecutionService | Provides management functions for interactive computing tasks |
linkis-engineconn-common engine connector common module
Define the most basic entity classes and interfaces in the engine connector. EngineConn is used to create a connection session for the underlying computing storage engine, which contains the session information between the engine and the specific cluster, and is the client that communicates with the specific engine.
Core Service | Core function |
---|---|
EngineCreationContext | Contains the context information of EngineConn during startup |
EngineConn | Contains the specific information of EngineConn, such as type, specific connection information with layer computing storage engine, etc. |
EngineExecution | Provide Executor creation logic |
EngineConnHook | Define the operations before and after each phase of engine startup |
The core logic of linkis-engineconn-core engine connector
Defines the interfaces involved in the core logic of EngineConn.
Core class | Core function |
---|---|
EngineConnManager | Provide related interfaces for creating and obtaining EngineConn |
ExecutorManager | Provide related interfaces for creating and obtaining Executor |
ShutdownHook | Define the operation of the engine shutdown phase |
linkis-engineconn-launch engine connector startup module
Defines the logic of how to start EngineConn.
Core class | core function |
---|---|
EngineConnServer | EngineConn microservice startup class |
The core logic of the linkis-executor-core executor
Defines the core classes related to the actuator. The executor is a real computing scene executor, responsible for submitting user code to EngineConn.
Core class | Core function |
---|---|
Executor | It is the actual computational logic execution unit and provides a top-level abstraction of the various capabilities of the engine. |
EngineConnAsyncEvent | Defines EngineConn-related asynchronous events |
EngineConnSyncEvent | Defines EngineConn-related synchronization events |
EngineConnAsyncListener | Defines EngineConn related asynchronous event listener |
EngineConnSyncListener | Defines EngineConn related synchronization event listener |
EngineConnAsyncListenerBus | Defines the listener bus for EngineConn asynchronous events |
EngineConnSyncListenerBus | Defines the listener bus for EngineConn synchronization events |
ExecutorListenerBusContext | Defines the context of the EngineConn event listener |
LabelService | Provide label reporting function |
ManagerService | Provides the function of information transfer with LinkisManager |
linkis-callback-service callback logic
Core Class | Core Function |
---|---|
EngineConnCallback | Define EngineConn's callback logic |
linkis-accessible-executor can be accessed executor
Executor that can be accessed. You can interact with it through RPC requests to get its status, load, concurrency and other basic indicators Metrics data.
Core Class | Core Function |
---|---|
LogCache | Provide log cache function |
AccessibleExecutor | The Executor that can be accessed can interact with it through RPC requests. |
NodeHealthyInfoManager | Manage Executor's Health Information |
NodeHeartbeatMsgManager | Manage the heartbeat information of Executor |
NodeOverLoadInfoManager | Manage Executor load information |
Listener | Provides events related to Executor and the corresponding listener definition |
EngineConnTimedLock | Define Executor level lock |
AccessibleService | Provides the start-stop and status acquisition functions of Executor |
ExecutorHeartbeatService | Provides heartbeat related functions of Executor |
LockService | Provide lock management function |
LogService | Provide log management functions |