Fresh An integrated analytics experience is offered by Amazon CloudWatch and Amazon OpenSearch Service. AWS has announced a zero-ETL connectivity between Amazon CloudWatch and Amazon OpenSearch Service, as well as a new integrated analytics experience.
By eliminating data duplication and simplifying log data analysis and visualisation, this integration streamlines log management while cutting down on operational and technical expenses. While OpenSearch users may query CloudWatch logs in-place without establishing separate extract, transform, and load (ETL) pipelines, CloudWatch Logs clients now have access to two more query languages in addition to CloudWatch Logs Insights QL.
Businesses frequently want various analytics features for their log data. Because of CloudWatch Logs’ scalability and ease of use in centralising logs from all of their systems, apps, and AWS services, several teams choose it. For more sophisticated analytics and visualisations, others need Amazon OpenSearch Service.
In the past, maintaining distinct ingestion pipelines or developing ETL procedures was necessary for integration between these services. By removing this complication and delivering the capability of OpenSearch analytics straight to CloudWatch Logs without requiring a data copy, this new connection enables users to benefit from the best of both services.
OpenSearch SQL and OpenSearch Piped Processing Language (PPL) are now supported by Amazon CloudWatch Logs right within the CloudWatch Logs Insights panel. SQL allows you to correlate logs using JOIN and analyse data. For easy-to-understand log analytics, you may employ SQL functions like JSON, mathematical, datetime, and string functions. The OpenSearch PPL may also be used for data analysis, aggregation, and filtering. AWS CloudTrail, AWS WAF, and Amazon Virtual Private Cloud (VPC) are examples of pre-built, off-the-shelf dashboards for vended logs that you can access with a few clicks.
Without requiring the configuration of individual widgets or the creation of particular queries, these dashboards provide quicker monitoring and troubleshooting through visualisations, such as the analysis of flows over time, top talkers, megabytes, and packets moved over time. VPC flows over time, top talkers, network traffic data, web request trends in AWS WAF, and patterns of API activity in AWS CloudTrail may all be examined.
Additionally, OpenSearch Service customers may now run SQL and PPL, analyse CloudWatch logs using OpenSearch Discover, and construct indexes and dashboards immediately without the need for additional ingestion pipelines or ETL procedures, much like they can with Amazon Simple Storage (Amazon S3) data analysis.
What is amazon opensearch service?
Using the opensource OpenSearch software package, Amazon OpenSearch Service is a search and analytics service. It is intended to assist companies in managing substantial volumes of data generated by machines.
Highlights of Integration For users of CloudWatch:
- Capabilities for queries Use OpenSearch SQL and PPL queries directly in the CloudWatch Logs Insights dashboard to expedite log examination.
- Features for analytics Access pre-made, off-the-shelf dashboards for vended logs, including VPC, AWS WAF, and CloudTrail logs, with just a few clicks. Through visualisations for examining flows over time, top talkers, megabytes, and packets moved over time, these dashboards provide quicker monitoring and troubleshooting without requiring the configuration of individual widgets or the creation of specialised queries.
- How to begin using CloudWatch Set up the OpenSearch Service interaction with CloudWatch Logs. See the documentation for the Amazon CloudWatch Logs vended dashboard and query features for further details.
For users of OpenSearch Services:
- Zero-ETL integration: Without creating or managing ETL procedures, access and examine CloudWatch data straight from OpenSearch Service. Through streamlined data management and no data duplication, this connection removes the need for separate ingestion pipelines while lowering storage expenses and operational overhead.
- How to get started with OpenSearch Choose CloudWatch as a data source from OpenSearch Service when creating a data connection. Consult the Amazon OpenSearch Service Developer Guide for further details.
Regional availability and pricing
This integration is now accessible in AWS regions that support direct queries to Amazon OpenSearch Service. You may visit the Amazon CloudWatch Pricing and Amazon OpenSearch Service Pricing pages.