Persist the List of Files in an External Stage in Snowflake

In Snowflake, you can access files external to the Snowflake instance using something called a STAGE. A STAGE is a named object that points to a location such as AWS S3, Azure Storage Account or Google Cloud Storage Buckets, and contains the required authentication credentials to access the files in that storage location. To programmatically …

Persist the List of Files in an External Stage in Snowflake Read More »

Notion Agile Project Management Kanban Board Template

If you want a project to be successful and deliver what is required on time and within budget, your team has to follow the correct processes using adequate technology in order to track the progress of the project and ensure you have visibility on the status of all tasks and dependencies. There are many tools …

Notion Agile Project Management Kanban Board Template Read More »

Get the Iteration of a Weekday in a Month on a Virtual Calendar

There are many situations where you need to identify the iteration of a weekday within a given month. Most commonly though is for the identification of public holidays e.g., the 4th Thursday in November – Thanksgiving in USA. There are no built-in methods to get this information in any SQL-based platform I have used thus …

Get the Iteration of a Weekday in a Month on a Virtual Calendar Read More »

Dynamic SQL in Databricks and SQL Server

What is dynamic SQL? Dynamic SQL is a programming technique where you write a general purpose query and store it in a string variable, then alter key words in the string at runtime to alter the type of actions it will perform, the data it will return or the objects it will perform these actions …

Dynamic SQL in Databricks and SQL Server Read More »

Calculate Easter Sunday Dynamically using SQL

In any data warehousing or analytics solution, you will eventually need to generate a calendar table in order to perform time intelligence analytics. One key aspect that most businesses will require is to view KPIs by working days only i.e. days when the business was operating normally. Each business will have their own unique rule sets …

Calculate Easter Sunday Dynamically using SQL Read More »

Function to mount a storage account container to Azure Databricks

What does it mean to mount a storage account to Azure Databricks Databricks has a built in “Databricks File System (DBFS)”. It is a distributed file system mounted onto your Databricks workspace. It is mounted directly to your cluster and is only accessible while the cluster is running. It is an abstraction on object storage. …

Function to mount a storage account container to Azure Databricks Read More »

Get the latest file from Azure Data Lake in Databricks

There are many ways to orchestrate a data flow in the cloud. One such option is to have an independent process pull data from source systems and land the latest batch of data in an Azure Data Lake as a single file. The next layer where you process the data can be handled in many …

Get the latest file from Azure Data Lake in Databricks Read More »

Dynamic Datasets in Azure Data Factory

In a previous post linked at the bottom, I showed how you can setup global parameters in your Data Factory that is accessible from any pipeline at run time. This post will show you how you can leverage global parameters to minimize the number of datasets you need to create. Specifically, I will show how …

Dynamic Datasets in Azure Data Factory Read More »