Discovering and surfacing telemetry traditionally can be a tedious and challenging process, especially when it comes to pinpointing specific issues for debugging. However, as applications and pipelines grow in complexity, understanding what’s happening beneath the surface becomes increasingly crucial. A lack of visibility hinders the development and maintenance of high-quality applications and pipelines, ultimately impacting customer experience. Comprehensive observability tools are a must to empower developers and data engineers to quickly identify and resolve issues.
Introducing Snowflake Trail
Snowflake Trail is a rich set of Snowflake capabilities that allows developers and data engineers to observe and act on their applications and data pipelines through Snowsight or third-party tools. Leveraging Snowflake’s Query History, Event Tables, Alerts and Notifications as the telemetry foundation, Snowflake Trail provides enhanced visibility into data quality, pipelines and applications. Each of these signals empowers developers to monitor, troubleshoot and optimize their workflows with ease. Snowflake Trail builds upon the observability foundation already in Snowflake. You are probably familiar with the built-in observability, with capabilities such as Task History and Dynamic Tables observability. With this launch, we are expanding the scope of what and how you can observe with Snowflake.
Fig 1: Snowflake Trail, encompassing a set of observability experiences in Snowflake
Effortless telemetry with one simple setting
Snowflake Trail is built with automated telemetry; no agent or setup tasks are needed. A default Event Table (public preview soon) is in the Snowflake database of every account, removing the need to create and manage your own custom event table. Snowflake Trail eliminates the need for any agent installation, tedious setup or data export tasks, providing fast insights into application and pipeline performance. With just one simple setting, you can gain visibility into the performance of your Snowpark code and its resource usage, so you can quickly diagnose and debug your apps and pipeline development. Events are all within Snowflake with no need for additional data transfer.
Customer stories
Our customers have seen significant improvements in their application and pipeline development with Snowflake Trail’s capabilities.
“When working with Snowpark UDFs, some of the logic can become quite complex. In some instances, we had thousands of lines of Java code that needed to be monitored and debugged. With the new logging and tracing capabilities, we are able to investigate issues in our code or data much quicker and find performance issues much faster,” said Nick Pileggi, Principal Solutions Architect at phData Inc., in regards to migrating Spark and Hadoop applications to Snowpark.
“Event Tables have been invaluable, as we take our Snowflake Native App to market. By choosing to share events with us, our customers benefit from us being able to assist them without needing to manually extract diagnostic data and send it to us. We are excited to see new Snowflake Trail features like the Log Explorer, which will help us hone in on the relevant information even faster, and Trace Viewer, which will help us remove performance bottlenecks in our code,” said James Weakley, Snowflake Data Superhero and co-founder at Omnata, a Snowflake Native App available in Snowflake Marketplace.
Built-in observability experiences reduce time to detect (TTD) and time to resolution (TTR)
Snowflake Trail provides a comprehensive set of telemetry signals, including metrics, logs and span events, to give developers a deeper understanding of their applications and pipelines. Snowsight is where these signals are brought together to help developers debug and detect issues near instantly, thereby reducing TTD and TTR. Key capabilities include:
Snowpark metrics (private preview): Understand the CPU and memory consumption of your code in Snowpark (Python) stored procedures and functions, using the new Snowpark metrics. Support for other languages coming soon.
Automatic Python DataFrame tracing (private preview): Snowpark DataFrames allow developers to write queries in native Python. When you use DataFrames on Snowflake, those operations will now also appear on the trace view, allowing you to see the full execution of your pipeline.
User code profiler for Python (private preview): Developers can attach a profiler to their stored procedure to understand where the most compute time is spent and better optimize their Python execution.
Log attributes (public preview): Filter logs further; available for Java and JavaScript, Python support coming soon.
Serverless Alerts (public preview): The power and evaluation logic of alerts are now available with the cost and warehouse optimization of serverless capabilities.
Developers can visualize what’s going on with their pipelines and apps, and interact with logs, metrics and tracing directly within Snowsight using features like Log Explorer (Public Preview) to easily view and filter logs from Snowpark code.
Fig 2. Explore logs in your event table easily within Snowsight. Filter by users, event type, severity, language.
And distributed tracing experience (private preview) for Snowpark makes it easier to visualize and troubleshoot calls across objects.
Fig 3. Distributed tracing in Snowsight, enables you to easily spot performance bottlenecks in your code
Last but not least, Snowflake Trail also provides data-quality monitoring (general availability soon) as part of Snowflake Horizon. Customers get built-in data-quality solutions with out-of-the-box system metrics (such as null count) or custom metrics that they can define to monitor data quality. Data engineers and stewards can effectively monitor and report on degradation in data quality across their organization.
Simply use Snowsight or Bring Your Own Tools: You can use Snowsight to monitor and trace pipelines, apps and resource usage directly in Snowflake. Even better, Snowflake Trail adheres to the industry-standard OpenTelemetry specification and notification destinations, allowing easy integration with your favorite observability and customizable notification tools, including Datadog, Grafana, Observe, Metaplane, PagerDuty, Slack and Microsoft Teams.
Get started with Snowflake Trail
Snowflake Trail marks a significant milestone in Snowflake’s observability journey, addressing the long-standing observability challenges faced by our users. With its rich telemetry, built-in observability experiences and easy integration with third-party tools, Snowflake Trail is poised to revolutionize the way developers build, deploy and maintain applications and pipelines in Snowflake. Learn more about Snowflake Trail and take it for a spin.