datalake

Function to mount a storage account container to Azure Databricks

What does it mean to mount a storage account to Azure Databricks Databricks has a built in “Databricks File System (DBFS)”. It is a distributed file system mounted onto your Databricks workspace. It is mounted directly to your cluster and is only accessible while the cluster is running. It is an abstraction on object storage. …

Function to mount a storage account container to Azure Databricks Read More »

Get the latest file from Azure Data Lake in Databricks

There are many ways to orchestrate a data flow in the cloud. One such option is to have an independent process pull data from source systems and land the latest batch of data in an Azure Data Lake as a single file. The next layer where you process the data can be handled in many …

Get the latest file from Azure Data Lake in Databricks Read More »

Dynamic Datasets in Azure Data Factory

In a previous post linked at the bottom, I showed how you can setup global parameters in your Data Factory that is accessible from any pipeline at run time. This post will show you how you can leverage global parameters to minimize the number of datasets you need to create. Specifically, I will show how …

Dynamic Datasets in Azure Data Factory Read More »