Skip to main content

Project Management

Feature​

The "Project Management" feature of StreamPark simplifies the management and deployment processes of Apache Flink and Apache Spark projects. By integrating with code repositories, users can automatically pull code, specify branches, and use Maven for automated Jar file builds. StreamPark provides CI/CD support, enabling automatic build and deployment when code updates occur. Additionally, users can submit jobs directly on the platform and monitor them in real-time, ensuring stable job execution. This feature enhances the automation, consistency, and security of project management, significantly improving development and deployment efficiency.

Project Management​

Apache Flink and Apache Spark offer programming models based on Java code for job development. For these types of jobs, users are required to write the program, build and package it to produce the target Jar, and then execute a command line (flink run or spark-submit) to submit the job. StreamPark offers project management capabilities, allowing users to easily add Apache Flink/Spark projects built by Java Maven, into StreamPark for management purpose. In this way, StreamPark will help package, build and manage the project, compile the target Jar, and after then user can add jobs, perform job submission and status monitoring, etc.

Project

How to use​

Click "Project Management" and add a new project. You will enter the project adding page. The operation is as follows:

Hot to use

Parameter Description:​

  • Project Name: (Required) Project name, cannot be repeated

  • Project Type: (Required) Project type, currently only supports Apache Flink

  • Repository URL：(Required) URL of the project repository, e.g: project's GitHub address or GitLab address

  • Password: (Optional) If the project requires a password to access

  • Branches: (Required) The project branch: Will be automatically loaded based on the Repository URL to select the optional branch.

  • Build Argument: (Optional) Project build parameters, The build parameters here are the standard parameters of Maven. For example, if you want to specify the profile as dev, then here is -Pdev, and other parameters can be inferred from this.

  • POM: (Optional) The location of the pom.xml file of the target module to be built. For example, if the StreamPark Quickstart project directory structure is as follows:

    Pom

    Now, if you only need to build the apacheflinksql_demo module, and you don't need to build any other modules. Then, you need to specify the relative path of the pom.xml file of the apacheflinksql_demo module, which is: quickstart-flink/quickstart-apacheflink/apacheflinksql_demo/pom.xml

Remind

In StreamPark, the project management module has a built-in Maven build capability, similar to Jenkins, but only supports projects built by Maven. The security verification and interception of dangerous parameters on parameter input (Build Argument) will by performed by StreamPark. If the user's project itself has some risks or vulnerabilities, the user needs to handle them by himself. Some risky operations, such as using the exec-maven-plugin plug-in in the user's project, whether these plug-ins will be executed, and what risks will be brought, depending on the user's project itself.