Amazon read data from a Kafka topic, transform the records, and write them to an Amazon S3 destination using the extract, traansform, and load (ETL) function provided by Amazon Kinesis Data Firehose

It offers a publish-subscribe messaging system that is both extremely scalable and fault-tolerant In order to capture streaming data, including click-stream events, transactions, IoT events, and application and machine logs, many AWS customers have adopted Kafka

Apache Kafka’s setup, scaling, and management are made easier using MSK You are free to concentrate on your data and applications because we take care of the infrastructure

The solution is code-free and serverless, meaning there is no server infrastructure to maintain Using the console, the data transformation and error-handling algorithms may be set up quickly

The data source is Amazon MSK, the data destination is Amazon S3, and the data transfer logic is handled by Amazon Kinesis Data Firehose

In case something goes wrong, it also handles the logic for errors and retries When a record cannot be processed, the system sends it to the S3 bucket of your choice for manual review

The Data Firehose delivery stream retrieves data from your MSK cluster, buffers it for a threshold size and time that you specify, and then uploads the single file of buffered data to Amazon S3

Data types can also be converted using Kinesis Data Firehose delivery streams The formats JSON to Apache Parquet and Apache ORC are supported by built-in transformations

For more details go to govindhtech.com