What does it mean to mount a storage account to Azure Databricks Databricks has a built in “Databricks File System (DBFS)”. It is a distributed file system mounted onto your Databricks workspace. It is mounted directly to your cluster and is only accessible while the cluster is running. It is an abstraction on object storage. …
Function to mount a storage account container to Azure Databricks Read More »
There are many ways to orchestrate a data flow in the cloud. One such option is to have an independent process pull data from source systems and land the latest batch of data in an Azure Data Lake as a single file. The next layer where you process the data can be handled in many …
Get the latest file from Azure Data Lake in Databricks Read More »
In a previous post linked at the bottom, I showed how you can setup global parameters in your Data Factory that is accessible from any pipeline at run time. This post will show you how you can leverage global parameters to minimize the number of datasets you need to create. Specifically, I will show how …
