Skip to main content
Version: 1.1.3

Pipeline Engine

This article mainly introduces the configuration, deployment and use of pipeline (>=1.1.0 version support) engine.

1. Configuration and deployment​

1.1 Version selection and compilation​

Note: before compiling the pipelineengine, you need to compile the linkis project in full Currently, the pipeline engine needs to be installed and deployed by itself

This engine plug-in is not included in the published installation and deployment package by default, You can follow this guide to deploy the installation https://linkis.apache.org/blog/2022/04/15/how-to-download-engineconn-plugin Or manually compile the deployment according to the following process

Compile separatelypipeline 

${linkis_code_dir}/linkis-enginepconn-pugins/engineconn-plugins/pipeline/
mvn clean install

1.2 Material deployment and loading​

The engine package compiled in step 1.1 is located in

${linkis_code_dir}/linkis-engineconn-plugins/engineconn-plugins/pipeline/target/out/pipeline

Upload to the engine directory of the server

${LINKIS_HOME}/lib/linkis-engineplugins

And restart the linkis engineplugin to refresh the engine

cd ${LINKIS_HOME}/sbin
sh linkis-daemon.sh restart linkis-cg-linkismanager

Or refresh through the engine interface. After the engine is placed in the corresponding directory, send a refresh request to the linkis CG engineconplugin service through the HTTP interface.

  • Interfacehttp://${engineconn-plugin-server-IP}:${port}/api/rest_j/v1/rpc/receiveAndReply

  • Request mode POST

{
  "method": "/enginePlugin/engineConn/refreshAll"
}

Check whether the engine is refreshed successfully: if you encounter problems during the refresh process and need to confirm whether the refresh is successful, you can view thelinkis_engine_conn_plugin_bml_resourcesOf this tablelast_update_timeWhether it is the time when the refresh is triggered.

#Log in to the database of linkis
select *  from linkis_cg_engine_conn_plugin_bml_resources

1.3 Engine label​

Linkis1.XIt is carried out through labels, so it is necessary to insert data into our database. The insertion method is shown below.

EngineConnPlugin Engine plug-in installation

2. Use of engine​

2.1 Task submission via linkis cli​

Link 1.0 provides cli to submit tasks. We only need to specify the corresponding enginecon and codetype tag types. The use of pipeline is as follows:

  • Note that the enginetype pipeline-1 engine version setting is prefixed. If the pipeline version is V1 , it is set to pipeline-1 
sh bin/linkis-cli -submitUser  hadoop  -engineType pipeline-1  -codeType pipeline  -code "from hdfs:///000/000/000/A.dolphin  to file:///000/000/000/B.csv"

from hdfs:///000/000/000/A.dolphin to file:///000/000/000/B.csv 3.3 Explained

For specific use, please refer to： Linkis CLI Manual.

becausepipelineThe engine is mainly used to import and export files. Now let's assume that importing files from a to B is the most introduced case

2.2 New script​

Right click the workspace module and select Create a new workspace of typestorageScript for

2.3 Script​

Syntax is：from path to path​

The syntax is file copy rule:dolphinSuffix type files are result set files that can be converted to.csvType and.xlsxType file, other types can only be copied from address a to address B, referred to as handling

#dolphin type
from hdfs:///000/000/000/A.dolphin to file:///000/000/000/B.csv
from hdfs:///000/000/000/A.dolphin to file:///000/000/000/B.xlsx

#Other types
from hdfs:///000/000/000/A.txt to file:///000/000/000/B.txt

A file importing script to B folder

from hdfs:///000/000/000/A.csv to file:///000/000/B/
  • from grammar，to：grammar
  • hdfs:///000/000/000/A.csv：Input file path
  • file:///000/000/B/： Output path

file B is exported as file A

from hdfs:///000/000/000/B.csv to file:///000/000/000/A.CSV
  • hdfs:///000/000/000/B.csv： Input file path
  • file:///000/000/000/A.CSV： Output file path

Note: no semicolon is allowed at the end of the syntax; Otherwise, the syntax is incorrect.

2.4 result​

speed of progress

historical information