What is Apache StreamPark™?

🚀 A magical framework that makes stream processing simpler!

In the realm of real-time processing, Apache Spark™ and Apache Flink® represent significant advancements, especially Flink, which is widely considered the next-generation big data stream processing engine. Through our experience with Apache Flink® and Apache Spark™, we discovered many parts of the programming model, startup configurations, and operation management that could be abstracted and shared. We solidified some of these excellent experiences and combined them with industry best practices. After relentless effort, we created: Apache StreamPark™.

screenshot

Apache StreamPark™ is an all-in-one stream processing platform. It standardizes project configurations, encourages functional programming, defines the best programming practices, and provides a series of out-of-the-box connectors. It standardizes the entire process from configuration, development, testing, deployment, monitoring, to operation and maintenance. It offers both Scala and Java interfaces and provides a one-stop platform for stream processing job development and management, supporting the full lifecycle from development to deployment. Using Apache StreamPark™ to develop stream processing jobs can significantly reduce the learning curve and development threshold, allowing developers to focus on core business logic.

Core Features

🎉 Application Development Scaffold

Apache StreamPark™ provides an easy-to-use development framework that enables developers to efficiently build stream processing applications. It supports multiple versions of Apache Flink® and Apache Spark™, lowering the learning cost and development threshold, and allowing developers to focus on core business logic.

mutiple_version

🎉 Out-of-the-box Connectors

Apache StreamPark™ offers a series of out-of-the-box connectors that simplify integration with various data sources and destinations. These connectors make data acquisition and processing more efficient, supporting various data scenarios.

connectors

🎉 One-stop Stream Processing Operations Platform

As an all-in-one stream processing operations platform, Apache StreamPark™ integrates development, debugging, deployment, and operational management functions. It provides a low-code platform via streampark-console, making it easier for users to manage Flink tasks and simplifying operations such as project compilation, publishing, and parameter configuration.

cicd

🎉 Supports Various Scenarios

Apache StreamPark™ supports a variety of application scenarios, including Catalog, OLAP, and Streaming Warehouse. This enables users to flexibly apply Apache StreamPark™ to meet the needs of real-time data analysis and processing.

🎉 ...

Architecture Design

🏳‍🌈 The core of Apache StreamPark™ consists of StreamPark Core and StreamPark Console:

StreamPark Archite

StreamPark Core

StreamPark Core is positioned as a development framework focused on coding. It standardizes configuration files and adopts a "convention over configuration" approach to development. It provides a development environment's RuntimeContext and a series of out-of-the-box connectors, extending methods related to DataStream. It integrates DataStream and Flink SQL API, simplifying operations, focusing on business logic, and improving development efficiency and experience.

StreamPark Console

StreamPark Console is a platform that integrates real-time data processing with low-code development. It efficiently manages Flink tasks and integrates functions like project compilation, publishing, parameter configuration, startup, savepoints, flame graphs, Flink SQL, and monitoring. It greatly simplifies the daily operation and maintenance of Flink tasks, incorporating many best practices. The platform enables projects, which were previously only accessible to large companies, to be available to everyone. Its goal is to create a real-time data warehouse and a unified big data solution for both stream and batch processing. The platform uses the following technologies (but not limited to):

Special thanks to these and many other excellent open-source projects, with the utmost respect!

Feel free to use this directly! Let me know if you'd like to tweak anything further.