make stream processing easier!!!
An amazing framework makes stream processing easier.
🚀 What is StreamPark
StreamPark is an easy-to-use stream processing application development framework and one-stop stream processing operation platform, Aimed at ease building and managing streaming applications, StreamPark provides scaffolding for writing streaming process logics with Apache Flink and Apache Spark.
StreamPark also provides a professional task management including task development, scheduling, interactive query, deployment, operation, maintenance, etc.
Apache Flink and Apache Spark are widely used as the next generation of big data streaming computing engines. Based on a bench of excellent experiences combined with best practices, we extracted the task deployment and runtime parameters into the configuration files. In this way, an easy-to-use RuntimeContext with out-of-the-box connectors would bring easier and more efficient task development experience. It reduces the learning cost and development barriers, hence developers can focus on the business logic. On the other hand, It can be challenge for enterprises to use Flink & Spark if there is no professional management platform for Flink & Spark tasks during the deployment phase. StreamPark provides such a professional task management platform, including task development, scheduling, interactive query, deployment, operation, maintenance, etc.
- Apache Flink & Spark application development scaffolding
- Supports multiple versions of Flink & Spark
- A range of out-of-the-box connectors
- one-stop stream processing operation platform
- Support catalog、olap、streaming-warehouse etc.
🏳🌈 Architecture of StreamPark
The overall architecture of StreamPark is shown in the following figure. StreamPark consists of three parts, they are StreamPark-core and StreamPark-console.
The positioning of
StreamPark-core is a framework uesd while developing, it focuses on coding development, regulates configuration files, and develops in the convention over configuration guide.
StreamPark-core provides a development-time RunTime Content and a series of out-of-the-box Connectors. Cumbersome operations are simplified by extending
DataStream-related methods and integrating DataStream and
Flink sql api .
development efficiency and development experience will be highly improved because users can focus on the business.
StreamPark-console is a comprehensive real-time
low code data platform that can manage
Flink tasks more convenient.
It integrates the experience of many best practices and integrates many functions such as project compilation, release,
parameter configuration, startup,
Flink SQL, monitoring, etc.,
which greatly simplifies the daily operation of Flink tasks and maintenance. The ultimate goal is to create a one-stop big data platform,
which can provide a solution that integrates flow and batch, and integrates lake and warehouse.
This platform uses technologies including but not limited to:
- Apache Flink
- Apache Spark
- Apache YARN
- Spring Boot
- Ant Design of Vue
- ANTD PRO VUE
- Monaco Editor
Thanks for the respect given by the above excellent open source projects and many unmentioned excellent open source projects