Logs and traces generated by applications are valuable sources of information that can help detect issues and improve performance.

However, they are often treated separately from other data, even though they are no different from the data an application works with.

In this tutorial, we will explore a different approach: treating logs and traces as part of a scalable cloud storage repository that can be analysed with the same techniques used for big data.

In this tutorial, we will send logs of Hazelcast Viridian Dedicated Clusters to Amazon OpenSearch.

• The Hazelcast Viridian Dedicated is a licensed version of the cloud-managed service. Dedicated means that you have a specific server (cloud instance) all dedicated to you, so there is no risk of interruption from other tenants. Hazelcast Viridian Dedicated is ideal for long-running, continuous production environments where workloads are generally predictable.
• Amazon OpenSearch Service to securely unlock real-time search, monitoring, and analysis of business and operational data. Amazon OpenSearch Service makes it easy to perform interactive log analysis, real-time application monitoring, website searches, and more.

Step 1: Set Up an Amazon OpenSearch Domain

• Create a Hazelcast Viridian Dedicated account: https://viridian.hazelcast.com/sign-in
• Create an AWS account: https://portal.aws.amazon.com/billing/signup#/start/email
• We recommend creating an IAM account for production use.
• From the list of regions on the top menu, select the region that’s closest to your Viridian cluster.
• Search for and select Amazon OpenSearch Service. The Dashboard opens ready for you to start the configuration.
• Click Create domain.

Step 2. Configure Your OpenSearch Domain

• In the Domain name field, enter a short name for your OpenSearch domain. For example, hazelcast-cloud-logs.
• In the Deployment type section:
• For testing purposes, select Development and testing as the deployment type.
• From the Version list, select Elasticsearch > 7.10.
• Select Include older versions.
• In the Data nodes section, under Availability Zones, select 1-AZ.
• In the Network section, select Public access to use fine-grained access control to secure the OpenSearch domain and data. VPC Peering connections between Viridian clusters and OpenSearch instances are not currently supported.
• In the Fine-grained access control section, select Create Master User and enter a username and password for the account.
• In the Access policy section, select Only use fine-grained access control.
• Click Create to build your domain. You’ll need to wait until your OpenSearch environment and the following URLs are created. You need both URLs to complete your logging setup.
• Kibana URL
• Domain endpoint

Step 3. Configure Logging Integration in Hazelcast Viridian

• Configure your Viridian Dedicated cluster to send log files to the OpenSearch domain.
• You’ll need the master user credentials and domain endpoint that you created when you configured OpenSearch.
• Sign into the Hazelcast Viridian console and select the cluster that want to integrate with OpenSearch.
• Select Manage > Logging.
• From the Logging Technology list, select Elastic Stack (ELK).
• In the Username and Password fields, enter your master user credentials.
• In the URL field, enter the domain endpoint for your OpenSearch domain followed by port number 443.
• For example: https://search--lxxxxxxx3f2v3wz3tlkh4.us-west-1.es.amazonaws.com:443
• OpenSearch supports HTTPS over port 443.
• Click Create Integration. Within 30 seconds, Hazelcast starts to send log files to OpenSearch, which are made available in Kibana.

Step 4. Set Up a Kibana Dashboard

• From your domain details page in OpenSearch, click the Kibana URL.
• Log in to Kibana, using your master user credentials.
• Select Explore on my own to skip onboarding.
• From the Select tenant dialog box, select Private and click Confirm.
• From the main menu, select Kibana > Discover to start your set up.
• Click Create index pattern. An index pattern tells Kibana which indices contain the data that you want to view.
• In the Index pattern field, enter a wildcard * to find all log data received from OpenSearch.
• When Kibana finds the Viridian cluster logs, click Next step.
• From the Time field list, select updated_at so that you’ll be able to filter your log file data by time.
• Click Create index pattern. Your log file data is ready to view. Simply select Kibana > Discover from the main menu.

Summary

In this tutorial, you've learned how to do the following:

• Create an Amazon OpenSearch domain to receive log files from your Viridian cluster.
• Configure your Viridian Dedicated cluster to automatically send log files to the OpenSearch domain.
• Load log files into a Kibana dashboard using an index pattern.

Finally, by keeping all the data together, we can apply machine learning models to detect situations of interest and alert us in real-time when unwanted behaviour is occurring or brewing.

This approach enables intelligent monitoring that goes beyond simple threshold-based alerts and can help identify complex issues that would otherwise go unnoticed.

We will discuss how to harness existing technologies to implement this approach, providing attendees with practical tips and insights that they can apply to their own projects.

You can join the Hazelcast Slack Community https://slack.hazelcast.com/ if you have any questions.