Analyze GCP Error Logs

Introduction

This workflow automatically analyzes Google Cloud Platform (GCP) log entries for errors over the past day and generates a detailed HTML report. It leverages the GCP logging API, authenticates with a service account, and uses an AI model to cluster error logs by root cause and provide actionable insights.

It performs the following tasks:

1. Runs on a daily schedule at 9:20 AM Berlin time.
2. Creates a JSON Web Token (JWT) to authenticate with the Google OAuth 2.0 system.
3. Requests an access token using the generated JWT.
4. Queries the GCP Logging API for error log entries from the past 24 hours filtered by severity.
5. Sends the log entries to an AI model that analyzes and clusters errors by root cause, summarizes the data, identifies affected services, and offers recommendations.
6. Processes the structured response from the AI and generates an HTML report with detailed tables and links to logs.
7. Outputs a structured, easy-to-read report for developers and operations teams to identify and respond to issues.

What you need

1. A GCP project with logging enabled.
2. A service account JSON key with permissions to read logs and generate OAuth tokens.
3. The private key and client email of the service account available as input.
4. Access to the Needle platform with the ability to create scheduled workflows.
5. An active Needle AI integration for the log analysis step.

How the flow works

Here is a breakdown of the nodes used in this workflow:

AI Analysis Capabilities

The AI model is specially prompted to extract the following insights:

1. Group logs by root cause.
2. Summarize main error patterns.
3. List affected services.
4. Analyze error timing.
5. Provide prioritized recommendations.

Output

At the end, you get a comprehensive HTML report detailing:

1. Report date and total error count.
2. Executive summary of main error patterns and affected services.
3. A table of error clusters with descriptions, sample errors, affected services, and log links.
4. Service-level error counts and top clusters.
5. Timeline analysis describing error spikes.
6. Prioritized recommendations for addressing the identified issues.

This report helps development and operations teams quickly understand error trends and root causes in their GCP environment.

Notes

1. The service account provided must have sufficient permissions to read logs.
2. The AI analysis depends on accurate and comprehensive logs; ensure your GCP logging configuration captures relevant error information.
3. Be mindful of API quotas on GCP and Needle when running this workflow daily.
4. Customize the AI prompt or report formatting to fit your team specific needs.