In Snowflake, you can access files external to the Snowflake instance using something called a STAGE. A STAGE is a named object that points to a location such as AWS S3, Azure Storage Account or Google Cloud Storage Buckets, and contains the required authentication credentials to access the files in that storage location. To programmatically …
Persist the List of Files in an External Stage in Snowflake Read More »
There are many situations where you need to identify the iteration of a weekday within a given month. Most commonly though is for the identification of public holidays e.g., the 4th Thursday in November – Thanksgiving in USA. There are no built-in methods to get this information in any SQL-based platform I have used thus …
Get the Iteration of a Weekday in a Month on a Virtual Calendar Read More »
What is dynamic SQL? Dynamic SQL is a programming technique where you write a general purpose query and store it in a string variable, then alter key words in the string at runtime to alter the type of actions it will perform, the data it will return or the objects it will perform these actions …
What does it mean to mount a storage account to Azure Databricks Databricks has a built in “Databricks File System (DBFS)”. It is a distributed file system mounted onto your Databricks workspace. It is mounted directly to your cluster and is only accessible while the cluster is running. It is an abstraction on object storage. …
Function to mount a storage account container to Azure Databricks Read More »
There are many ways to orchestrate a data flow in the cloud. One such option is to have an independent process pull data from source systems and land the latest batch of data in an Azure Data Lake as a single file. The next layer where you process the data can be handled in many …
Get the latest file from Azure Data Lake in Databricks Read More »
Why do you need a date table? In order to work with Time Intelligence Functions in Power BI, you must have a dedicated Date Table in your model.Time Intelligence is the mechanism by which Power BI can calculate measures such as Month To Date, Year To Date etc. A Date Table is one that meets the …
When would you work with JSON Data? JSON is a popular data representation format used on the web to exchange information between remote parties.It is also used for storing unstructured data in log files or NoSQL Document Databases such as MongoDB or Azure CosmosDB.SQL also has the ability to store JSON data in a text …
Query JSON data in SQL Server and Synapse Analytics Read More »
Overview Azure Synapse Analytics Serverless SQL Endpoint has the capability to query files that are stored in an Azure Data Lake using T-SQL code as if they were regular tables in a relational database.These files can be semi-structured or unstructured in nature. Using the Create External Table As Select (CETAS) functionality, you can even generate new …
Query Azure Data Lake via Synapse Serverless Security Credentials Setup Read More »
What is an External Table? This article will focus on the Synapse Analytics implementation of External Tables.However, note that there are other flavours of external tables and they behave slightly differently depending on which product you are using to defined it. SQL Server SQL Database Azure Synapse Analytics Analytics Platform System (PDW) External Tables in …
Dynamic External Tables in Azure Synapse Analytics On-Demand Read More »
TL;DR When defining your PySpark dataframe using spark.read, use the .withColumns() function to override the contents of the affected column. Use the encode function of the pyspark.sql.functions library to change the Character Set Encoding of the column. import pyspark.sql.functions dataFame = ( spark.read.json(varFilePath) ) .withColumns(“affectedColumnName”, sql.functions.encode(“affectedColumnName”, ‘utf-8’)) Scenario The scenario where this would be needed …
