By: KiranKumar Earalli, Principal-Data Engineering, LTIMindtree

Data Cloud – An essential for organizations of today

Modern-day enterprises, based on the cloud, collect and store copious amounts of data related to their businesses. Analyzing this data helps them understand competition, gain insights, and make data-driven decisions to stay relevant. With time, the amount of such data only grows, making data clouds essential and inevitable. According to a study by Statista, as of 2022, over 60% of all corporate data is stored on the cloud. A major advantage of adopting the cloud is that it offers a single location for storing all the data for easy access. Organizations can access it for daily analysis to reduce costs and optimize operations. However, these solutions come at a price, and managing costs effectively is a challenge that most organizations face.

Snowflake – A popular data cloud solution

Snowflake is a fully-managed platform that offers storage, computing, and cloud services. It operates on a consumption-based costing setup where users are charged in credits. It is crucial to understand the costing setup of the platform, including the implicit and hidden costs. Understanding, monitoring, and controlling them helps utilizing the platform optimally.

Fig. 1: The Snowflake Platform

While adopting Snowflake the finer details regarding costs are usually overlooked, because of lack of awareness. Unlike an on-premises data center, available at a yearly charge, a cloud-based data center charges based on the data volume and computing time. This pay-as-you-go model may seem like an optimum arrangement, but in fact, it often leads to unnecessary costs and consumption of valuable credits. Although it means that you pay for what you use, you may end up paying (even heavily) for what you don’t use efficiently. For example, a long-running bad query, which has failed, is discovered only at the end of a billing cycle. Until then, the bad query has already consumed precious credits for a significant amount of time even in the next cycle. Such inefficiencies may not have a significant impact on the cost in an on-premises environment, but on the cloud, it is a great price to pay.

Before we delve into the various aspects of cost management, let’s understand the factors that influence Snowflake costs significantly.

Fig. 2: Key factors impacting Snowflake costs

Keeping your Snowflake account costs in control and governing them requires efficient utilization of the platform. To ensure this, adopting a FinOps Framework at an organizational level proves extremely beneficial. FinOps, also known as “Cloud Financial Operations,” is a “financial management discipline, which helps engineering, finance, technology, and business teams to collaborate on data-driven decisions.” Bringing in this discipline brings accountability to the spending on cloud.

LTIMindtree helps incorporate this discipline for Snowflake cost management and governance, with its PolarSled FinOps solution.

PolarSled FinOps – The Science of Cost Management

As a recognized Snowflake partner, LTIMindtree has developed a framework, as a part of the PolarSled governance solution. The PolarSled FinOps framework helps manage costs across all of Snowflake’s storage, computing, and cloud services accounts. It disciplines expenses by identifying the areas of expenditure and helps control and optimize them.

What does PolarSled FinOps do?

The solution is a part of the PolarSled governance toolkit, designed to sustain operations and optimize expenditure. It helps estimate, monitor, control, and optimize your spending on the Snowflake account. Let us learn more about each of these aspects of the solution.

Cost Estimation

This is a crucial step in estimating the total cost, where we understand the kind of workloads being run and calculate the credits likely to be consumed. It helps build a solid business case for your Snowflake use case. PolarSled provides a realistic estimate of costs based on all implicit and explicit components. To begin with, this step involves understanding the granular details of the Snowflake cost model and mapping the workload. Based on these factors, the cost of Snowflake cloud services, storage, and computation is estimated to arrive at a total cost of ownership.

Cost Monitoring

The tool provides a multi-dimensional view of the costs, by classifying them based on factors like account, users, consumption type, virtual warehouse, etc. It is designed with cognitive capabilities and features over sixty dashboards, which give a 360-degree understanding of cost consumption. It also aids with predicting future costs by providing insights, such as approximate cost per query, and segregating users in high, medium, or low-risk zones. Monitoring costs is a cyclical process, which can be automated to control and optimize resources. It involves setting budgets and measuring compliance based on a set of KPIs, using automated tools. Budgets and targets can be adjusted as per the consumption pattern and the value generated.

Cost Control

Users can apply automated controls using this tool, to ensure that the costs stay within the defined limits. It also highlights the parameters in the account that do not match the cost goals recommended by Snowflake. The process starts with setting the budget limits per team, application, or line of business. Cost zones that require automated actions are then identified and resource monitors are mapped to these zones. Account-level properties are set up to ensure cost efficiency and the resource monitors and account settings are adjusted as per the requirement.

Cost Optimization

The optimization aspect involves continuous tracking and analyzing of queries. This helps to identify and remediate inefficient queries quickly and save costs. Optimization begins with setting up the right cost design and tuning all the layers of Snowflake for maximum efficiency. A dedicated organization, focused on cost optimization, is set up and an end-to-end process is defined to operationalize costs. The entire process is automated to ensure compliance.

Cost Best Practices

The framework provides cost management best practices and benchmarking as a part of the PolarSled FinOps solution. Our engineers and architects, with practical experience in several implementations on the Snowflake platform, have defined these practices and benchmarks. The tool automates the assessment of the Snowflake platform against the benchmarked best practices. It generates a health report at regular intervals and can provide persona-based dashboards and key event alerts.

Benefits of the PolarSled FinOps Solution

Managing costs with this solution ensures efficient usage and curb unnecessary spending. The PolarSled FinOps framework automatically recommends cost savings and remediation measures, and the intelligent optimization process helps reduce costs and enhance revenue.

Cost benefits

• Helps with estimating your Snowflake costs
• Monitors the Snowflake account’s cost parameters
• Helps control credit utilization by warehouses
• Aids with optimizing queries to reduce overall costs

Revenue benefits

• Increases overall returns on investment (ROI)
• Lowers overall expenditure on Snowflake
• Maximizes the value of your Snowflake data cloud investment

This solution has helped organizations across various industries make the most of their Snowflake cloud services while keeping their expenditures optimum. Our clients include leaders in the banking, financial services, FMCG, and manufacturing industries. Leveraging the capabilities of the PolarSled FinOps framework, they have been able to utilize the reusable FinOps components effectively. The greater win has been the ability to optimize their Snowflake governance costs by 30% and their overall costs by 10 – 20%.