By: KiranKumar Earalli, Principal-Data Engineering, LTIMindtree

Snowflake started in 2012 as a Data Warehouse (DW), and today it has evolved into a data cloud by supporting various workloads above and beyond DW. As part of recent innovations, the Snowflake platform has started supporting Snowpark, which, along with Spark programming constructs like Java/Scala/Python, can help achieve semi-structured and unstructured data processes, data science, etc. This blog is about how Snowpark eliminates challenges we face with Spark and how it simplifies your current data analytics platform with unified governance, scalability, and cost optimization.

Background

We have witnessed data growth over the last three to four decades, from file systems to Relational Database Management Systems (RDBMS) to data warehouses to data lakes to the current world of the data cloud. But if you look closely, you’ll see that most individuals, at least in the data field, are much more at ease with SQL-like syntax than with any other programming languages. But can all types of logic be achieved using SQL? Absolutely not. We need specific programming languages like Python, Java, Scala, etc., to accomplish a few different business requirements, such as data science and unstructured data processing. Spark has been the de facto distributed data processing system for the past ten years. Initially, Spark only supported Java and Scala, and although many data engineers at that time were not comfortable with these technologies, down the line, Pyspark was introduced. Again, it would be wrong to say everything can be accomplished using just Spark, as a data platform also requires data persistence, data governance, metadata management, etc.

Challenges with Spark

Spark, a distributed data processing system, has come a long way in a decade, but at the same time, we have a lot of challenges associated with developing and maintaining Spark code or clusters even today. A few of them are listed here.

Key Challenges of Spark

• Lack of talent in the industry
• Learning curve is long
• Need highly skilled talent to maintain the Spark cluster
• Ad hoc analysis on Spark is difficult. You need to know the requirements beforehand so that data is partitioned accordingly for read/write optimization.
• No centralized metadata layer for optimization
• Spark is based on Scala, so we cannot achieve sub-second responses as of now
• Need additional tools for Data Governance
• Need for strong administration of on-prem Spark cluster
• Spark performance tuning process is tedious
• No result cache, so even if the same results are required, the entire processing should take place

What is Snowpark?

As mentioned earlier, not everything can be achieved by SQL in the Snowflake platform, so there was a strong need to introduce support for distributed processing engines like Spark. As Spark was a de facto standard for big data processing, Snowflake revolutionized it with Spark API at the front. Behind the scenes, most of it got converted to SQL, which took advantage of the cloud services layer and increased current Spark jobs’ performance by at least two to ten times. With Snowflake’s introduction of Snowpark, the life of data engineers became quite easy as most business use cases were developed and deployed within Snowflake.

Snowpark is the client-side API in Java/Scala/Python, which provides developers with a similar experience to Spark in terms of coding. But behind the scenes, Snowpark uses all the features of Snowflake to achieve high performance. Now that programming construct is available in Snowpark, we can also develop any business logic that couldn’t be handled with SQL, like parsing unstructured data like PDF, images, audio files, etc.

So with Snowpark, we have a unified, comprehensive data platform.

Fig: Snowpark Internal Architecture

Advantages of Snowpark

The introduction of Snowpark provides us with the following advantages:

• Easy scalability
• Support for unstructured data processing
• Developers can have the same experience as Spark from the code perspective
• Out-of-the-box performance
• Centralized data governance
• Easier integration with third-party libraries of Python
• No hassle of maintaining a Spark/Hadoop cluster
• No hassle of upgrading the Hadoop cluster

Why is Snowpark a game changer?

As mentioned earlier on the Spark challenges, most of them on the administration, performance, and cost are solved quickly with Snowpark.

As we are already aware of Snowflake features, its architecture, out-of-the-box performance, near zero maintenance, etc., most of the features can be directly used by Snowpark as it internally converts the Data Frame code to SQL and does push-down optimization. A few key benefits, in addition to the ones already captured above:

• Developers can still enjoy the data frame style of coding without worrying about how it’s running in the background
  • Not only do developers enjoy the programming construct, but they also don’t have to worry about partitioning based on read patterns of data, which is a critical aspect while processing in Spark for optimal performance.
• Clients get the benefit of not maintaining a separate cluster for Spark
  • No need for the administration of Cluster
  • Automatic scalability
  • Security patch updates are not required.
  • Need not worry about tedious platform upgrades.
• Faster insights as Snowpark runs much faster compared to Apache Spark
  • Partition pruning: Snowflake, based on metadata, automatically knows which micro-partitions need not be read for the computation to achieve better performance.
  • Result cache: If the same query is run multiple times, Snowflake doesn’t recompute unless the underlying data changes or the 24-hour window has been exceeded. This saves a lot of computing costs.
  • Access to near-unlimited resources at the click of a button.
• Moving from a 24*7 dedicated Spark cluster to a consumption-based pricing model of Snowflake
  • No hassle of capacity planning
  • No need to buy hardware for the peak loads
  • No resource contention issues; each business unit can access the same data without being dependent on other jobs.
• Simplification of the technology with just one platform for all the needs
  • Near zero administration
  • No hassle of integrating multiple technologies to solve business problems
  • No hassle of compatibility issues between different technologies
  • Easily create multiple environments with zero copy-cloning
  • Better way to handle Continuous Integration (CI)/Continuous Deployment (CD)
  • Easier way to handle unit test cases
  • Majority of the issues are caught during compile time as compared to SQL’s runtime issues
• Unified governance
  • One place to handle user access Role Based Access Control (RBAC) model
  • Easy way to perform data masking
  • Complete compliance with General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPPA), California Consumer Privacy Act(CCPA), etc.
• Ready for next-gen use cases like data sharing in near real-time, AI/ML implementation, etc.
• Cloud agnostic solution means as the need arises, the same solution can be ported to any of the major hyperscalers (AWS, Azure, and GCP)

Migration to Snowpark in a faster, cheaper & de-risked manner

Now that you know that Snowpark is the way to go, you would want to migrate from Spark to Snowpark. It is important to analyze your current code to identify the following:

• If it’s the right fit for Snowpark migration
• Risks associated with automation, if any
• Accuracy of the automated conversion
• Need for data migration to Snowflake
• Reconciliation of the data between the source and target
• Governance and security of the Snowflake platform

Conclusion

To conclude, Snowpark is a game-changer. Organizations can enjoy the benefits of Scala/Python programming constructs simultaneously. They can further reap benefits from tons of features that Snowflake offers, such as time travel, zero-copy clone, secure data sharing, consumption-based pricing, easy scale-up or scale-out, automatic partition pruning, etc. Snowpark can easily deliver performance improvements, eliminate infrastructure maintenance costs and effort, and reduce operational costs, as you pay based on usage. Snowpark will also eliminate the need for multiple platforms or technologies for data engineering, simplifying the overall data analytics solution. As one of the elite and strategic Snowflake partners, LTIMindtree was part of the Snowpark accelerated program and got an early preview of Snowpark. Read more about our accelerator-PolarSled.