Digitization is advancing in almost all industries. In the age of Big Data, the amount of data to be analyzed is increasing exponentially. With the movement of IoT, more and more data is being produced that must be interpreted by various applications in heterogeneous system landscapes. In the process, evaluations and analyses of the enormous amounts of data in the context of reports reach their limits.
This development brings the so-called "event-driven architecture" to the fore. The goal is not the periodic tapping and evaluation of large amounts of data, but the near-real-time processing of events as they occur. These events can be real-world events (IoT), but can also represent transactional processes from SAP systems.
CENTRAL DATA SOURCE
The usual structure of interfaces developed specifically for individual applications, systems or plants, tailored to individual requirements and thus unique, lead in reality to a multitude of isolated solutions that are basically similar, but differ in terms of data formats, selection or aggregation logic. Apache Kafka provides a platform to unify all events from different systems and to map and historize a continuous data flow of events in real-time.
By decoupling systems, testing efforts are minimized and resilience is maximized. Data can be consumed, aggregated, grouped and provided with other logics and then provided again in an adapted form.
FUNCTIONING OF KAFKA
An event represents a record of something that has just happened and can also be referred to as messages. These messages are sent by the so-called "producer" to one or more KAFKA topics and persisted there. The topics represent related events and can be freely defined. In the SAP context, for example, all goods movements from different SAP systems can be stored in a topic. An event always consists of a timestamp, a key and a value, whereby the latter is unlimited and can contain entire tables. The data is managed by brokers and saved on different partitions. By mirroring data on different brokers the topic is secured by failures.
As buyers of the data stand the so-called "consumers". These are completely decoupled from the "producer" side and can process the events asynchronously. Several consumers can also be on different processing states, whereby each consumer knows its own state. A consumer can read, interpret and process the data from one or more topics. Processing logics can be defined on the producer as well as on the consumer side.
Integration with SAP ERP
For real-time messaging, data integration and data processing on a large scale, there is enormous potential for SAP ERP (ECC and S/4HANA), but also for most other products from the extensive SAP portfolio. In order to implement a modern as well as scalable approach to the integration of Apache Kafka and SAP, there are different options. Options include RFC, BAPI, IDOC, SOAP and OData as well as KAFKA Connect. For example, RFC modules can also be implemented to transfer data to Kafka as part of the update process.
Confluent offers predefined solutions here for SAP, with which SAP databases can be easily accessed. At the same time, the Kafka Connect tool offers options for storing events in other databases, which can later be used for evaluation or third-party systems and thus no longer need to burden the SAP system.
Events can also be consumed by other Topics to drill down and evaluate the relevant data as the events occur. By unifying all systems and data sources in a common platform, a wide variety of data can be transformed into completely new structures with new added values and in turn be made available as a Topic.
ADVANTAGES OF EVENT-BASED BUSINESS WORKFLOWS
Effects of a non-event based approach:
- Synchronous waiting for all work to be completed before sending response.
- Dependencies of the systems due to updates or releases
- Low transparency of dependencies
- slow and complex development process due to dependencies
- higher test effort due to organization and orchestration of tests
- redundant data transfer and redundant connection to a data lake / BW
- many requests can overload the source system
Impact of the event-based concept:
- Respond to the user much earlier in the process
- systems decoupled from each other (in technical terms)
- system-specific advantages of direct communication are lost (e.g. SAP RFC calls)
- data may have to be converted more than once
ADVANTAGES FOR THE SOURCE SYSTEM AND THE TARGET ARCHITECTURE
Event-based data communication brings significant benefits for both the source system and the target architecture:
The source system's program logic is greatly simplified and dependencies on downstream systems are eliminated. Changes to the source system are required less frequently, and only when its own coherent functionality is affected. As a result, testing and deployment become much easier.
The target architecture gains greater freedom to change when no other system is dependent on it. Transparency of impact on the system and data is significantly higher, and autonomy provides agility and flexibility.
CONSILIO's experts provide the necessary know-how for the introduction of Kafka, support you in minimizing the programming effort, and create use cases for the use of Kafka together with the users.
In the event-driven architecture, decoupling the systems minimizes the test effort and maximizes reliability. Data can be consumed, aggregated, grouped and provided with other logics and then provided again in an adapted form.
