Skip to main content
Version: Next(1.7.0)

EngineConn Design

1. Overview

EngineConn: engine connector, which is used to connect to the underlying computing and storage engine to complete task execution, task information push and result return, etc. It is the basis for Linkis to provide computing and storage capabilities.

2. Overall Design

 The overall design idea of EngineConn is to complete the acquisition and storage of the session information of the underlying engine when starting, complete the connection between the EngineConn process and the underlying engine, and then complete the scheduling of tasks to the underlying engine Session stored in EngineConn through the Executor unit for execution. and get execution-related information.

2.1 Technical Architecture

Introduction to related terms:

EngineConn: Used to store the session information of the underlying engine. To complete the connection with the underlying engine, for example, the Spark engine stores the SparkSession.

Executor: The scheduling executor used to accept the task passed by the caller (such as: Entrance), and finally submit the task to the underlying engine Session for execution. Different tasks implement different Executor classes. The most used is the interactive ComputationExecutor, which is used to accept tasks and push task information to the caller in real time. And the non-interactive ManageableOnceExecutor that accepts only one task is used to complete the submission and execution of the task started by EngineConn.

arc

2.2 Business Architecture

Component nameFirst-level moduleSecond-level moduleFunction points
LinkisEngineConnlinkis-engineconn-commonThe common module of engine conn, which defines the most basic entity classes and interfaces in engine conn.
LinkisEngineConnlinkis-engineconn-coreThe core module of the engine connector, which defines the interfaces involved in the core logic of EngineConn.
LinkisEngineConnlinkis-executor-coreThe core module of the executor, which defines the core classes related to the executor.
LinkisEngineConnlinkis-accessible-executorThe underlying abstraction of the accessible Executor. You can interact with it through RPC requests to obtain its status, load, concurrency and other basic indicators Metrics data
LinkisEngineConnlinkis-computation-engineconnRelated classes that provide capabilities for interactive computing tasks.

3. Module design

Input: The caller executes the task

Output: return task information such as execution status, results, logs, etc.

Key logic: the timing diagram of the key logic of task execution

time

Key Notes:

  1. If it is a serial Executor, after EngineConn receives a task, it will mark EngineConn as Busy and cannot accept other tasks, and will judge whether the lock of the task is consistent to prevent EngineConn from being submitted by multiple callers at the same time. After the task is executed, it becomes the Unlock state
  2. If it is a parallel Executor, after EngineConn receives the task, the state is still in the Unlock state, and it can continue to accept the task. Only when the number of concurrent tasks is reached or the machine index is abnormal will it be marked as Busy state
  3. If it is an Once type task, EngineConn will automatically execute the task after it is started, and the EngineConn process will exit after the task is executed.

4. Data structure/storage design

not involving

5. Interface design

Brief introduction of other classes:

The common module of linkis-engineconn-common engine connector defines the most basic entity classes and interfaces in the engine connector.

Core ServiceCore Function
EngineCreationContextcontains the context information of EngineConn during startup
EngineConncontains the specific information of EngineConn, such as type, specific connection information with layer computing storage engine, etc.
EngineExecutionProvides the creation logic of Executor
EngineConnHookDefines the operations before and after each stage of engine startup

The core module of linkis-engineconn-core engine connector defines the interfaces involved in the core logic of EngineConn.

Core ClassesCore Functions
EngineConnManagerProvides related interfaces for creating and obtaining EngineConn
ExecutorManagerProvides related interfaces for creating and obtaining Executor
ShutdownHookDefines actions during engine shutdown
EngineConnServerStartup class of EngineConn microservice

linkis-executor-core is the core module of the executor, which defines the core classes related to the executor. The executor is the real computing execution unit, which is responsible for submitting user code to EngineConn for execution.

Core ClassesCore Functions
Executoris the actual computing logic execution unit, and provides top-level abstraction of various capabilities of the engine.
EngineConnAsyncEventdefines EngineConn related asynchronous events
EngineConnSyncEventdefines the synchronization event related to EngineConn
EngineConnAsyncListenerdefines EngineConn-related asynchronous event listeners
EngineConnSyncListenerdefines EngineConn-related synchronization event listeners
EngineConnAsyncListenerBusDefines the listener bus for EngineConn asynchronous events
EngineConnSyncListenerBusDefines the listener bus for EngineConn sync events
ExecutorListenerBusContextdefines the context of the EngineConn event listener
LabelServiceProvide label reporting function
ManagerServiceProvides the function of information transfer with LinkisManager

linkis-accessible-executor: The underlying abstraction of the Executor that can be accessed. You can interact with it through RPC requests to obtain basic metrics such as its status, load, and concurrency.

Core ClassesCore Functions
LogCacheProvides the function of log caching
AccessibleExecutorAn Executor that can be accessed and interacted with via RPC requests.
NodeHealthyInfoManagerManage Executor's health information
NodeHeartbeatMsgManagerManage Executor's heartbeat information
NodeOverLoadInfoManagerManage Executor load information
Listener-relatedProvides events related to Executor and corresponding listener definitions
EngineConnTimedLockDefine Executor level lock
AccessibleServiceProvide the start-stop and status acquisition functions of Executor
ExecutorHeartbeatServiceProvides Executor's heartbeat-related functions
LockServiceProvides lock management functions
LogServiceProvides log management functions
EngineConnCallbackDefine the callback logic of EngineConn

Related classes that provide capabilities for interactive computing tasks.

Core ClassesCore Functions
EngineConnTaskdefines interactive computing tasks submitted to EngineConn
ComputationExecutordefines an interactive Executor, which has interactive capabilities such as status query and task kill, and can only execute tasks once by default.
ConcurrentComputationExecutorInteractive synchronous concurrent Executor, inherited from ComputationExecutor, but supports executing multiple tasks at the same time
AsyncConcurrentComputationExecutorInteractive asynchronous concurrent Executor, inherited from ComputationExecutor, supports multiple tasks to be executed at the same time, and the task does not occupy the execution thread and adopts the form of asynchronous notification
TaskExecutionServiceProvides management functions for interactive computing tasks

6. Non-functional design

6.1 Security

  1. All the relevant information of the task can only be queried by submitting the user
  2. The default startup user of the EngineConn process is the submission user

6.2 Performance

EngineConn that supports concurrency supports colleagues to run a large number of tasks concurrently. For example, a single Trino EngineConn can run more than 300 trino tasks at the same time

6.3 Capacity

not involving

6.4 High Availability

EngineConn is a process started on demand and task. Support high availability

6.5 Data Quality

not involving