The popular open-source event store and stream processing platform Apache Kafka has developed into the industry standard for data streaming. With IBM Event Streams on IBM Cloud, a fully managed Kafka service, developer Michael Burgess offers an understanding of schemas and schema management to enhance your event-driven applications.
Define a schema
A schema explains how data is organized.
For case:
A basic Java class that models a product order from an internet retailer might include the following fields at the beginning:
public class Order{ private String productName private String productCode private int quantity […] }
A schema like this Avro schema might be used to define the structure of records created by order objects formed with this class and submitted to a Kafka topic:
{ “type”: “record”, “name”: “Order”, “fields”: [ {“name”: “productName”, “type”: “string”}, {“name”: “productCode”, “type”: “string”}, {“name”: “quantity”, “type”: “int”} ] }
Why is using a schema recommended?
Apuana Kafka transmits data without first verifying the contents of the messages. It is unable to see what kinds of data are being sent or received or what kinds of data it might contain. Kafka doesn’t go into your messages’ metadata.
Decoupling producing and consuming applications so they interact through a Kafka topic rather than directly is one of Kafka’s features. As a result, they can each operate at their own pace. Nevertheless, they must decide on the same data format; otherwise, the consuming applications won’t be able to reassemble the data that they get into a meaningful form. The apps must all operate under the same presumptions regarding the data’s structure.
Within the context of Kafka, a schema explains the data structure of a message. It outlines the types of fields that are required in every message as well as the fields that must be present.
In other words, a schema creates a clear contract between a producing application and a consuming application, enabling the latter to accurately parse and understand the contents in the messages they get.
A schema registry: what is it?
By acting as a repository for managing and validating schemas inside your Kafka cluster, a schema registry helps your cluster. It serves as a database to hold your schemas and offers an interface for retrieving and managing schemas. The evolution of schemas is also verified by a schema registry.
Use a schema registry to make your Kafka system more efficient
In your Kafka context, a schema registry is effectively an agreement about the structure of your data. You may prevent typical mistakes in application development, like inconsistent data across your generating and consuming applications that could eventually result in data corruption, and poor data quality by maintaining a consistent store of the data formats in your applications. In addition to being technically essential, having a well-managed schema registry supports the strategic objectives of considering data as a valuable commodity and greatly facilitates the process of using data as a product.
By imposing guidelines for schema evolution, a schema registry improves the quality of your data and guarantees data consistency. Thus, a schema registry guarantees that your messages will continue to work together even if schema versions change over time, in addition to guaranteeing data consistency between generated and consumed messages. It is highly likely that during a firm’s existence, the messages that the apps supporting the business exchange will need to alter in format.
For instance, the Order class in the previously used sample schema might get a new status field; the product code field might be changed to a field that combines the department and product numbers, or something similar. As a result, the schema of the objects in our business domain is always changing, thus at any given time, you must be able to guarantee agreement on the schema of messages related to any given topic.
The evolution of schemas follows several patterns:
- Forward Compatibility: While awaiting migration to the new version of the schema, all consuming apps can continue to receive messages thanks to forward compatibility, which allows producing applications to upgrade to a new version of the schema.
- Backward compatibility refers to the ability of consuming programs to be moved to a new schema version ahead of generating applications, allowing the former to continue consuming messages created in the latter format.
- Full Compatibility: When a schema is fully compatible, it can function both forwards and backwards.
By guaranteeing the forward, backward, or full compatibility of new schema versions and restricting the introduction of incompatible schema versions, a schema registry can impose rules for schema evolution.
A schema registry, which offers an easy way to track and audit changes to your topic data formats, makes it easier to comply with data governance and data quality requirements by acting as a store of versions of schemas used inside a Kafka cluster, both past and present.
Next, what?
In conclusion, a schema registry is essential for controlling versioning, schema evolution, and data consistency in distributed systems, all of which help to ensure interoperability between various components. An Enterprise subscription for Event Streams on IBM Cloud includes access to a Schema Registry. Make sure your environment is optimized by taking advantage of this capability on IBM Cloud’s fully managed Kafka service to create intelligent, responsive apps that respond instantly to events.
