Parameterize Synapse Analytics Spark Notebooks Efficiently

When creating pipelines in any sort of data flow to move data from an incoming source to a target location, ideally you don’t want to create single-purpose activities. These would only perform one action or set of actions on a specific set of source and target objects. E.g. take the source.sales table, filter out where the date is NULL and import into the target.FactSales table in SQL. This dataflow is only ever useful for that specific sales table. The time it took to develop and test the dataflow cannot be utilized to perform work on other datasets as well.
This makes maintenance and extensibility of the data flow more difficult and time-consuming because there is ultimately “hardcoded” activities running. To integrate any additional datasets, you need to duplicate the code and make small adjustments.
To solve this problem, you would use parameterization of course.
In Synapse Analytics Notebooks, one accomplishes this using a parameters cell.
Tagging a cell in the notebook as the “Parameters Cell” is very straightforward, as shown below. However, it still has a big drawback. If you want to extend the notebook by adding more parameters in future, you would need to alter the Pipeline and the Activity calling the notebook to add the parameter reference as well. This adds extra tasks to the development flow and can easily be overlooked, causing unforeseen errors when testing the notebook.
To solve this, we can use a single JSON Object or JSON Array of Objects as the parameter instead. But first, lets tag the cell as the parameters cell and take it from there.

Tagging a cell in a notebook as the parameters cell

In Synapse Analytics, when calling a Notebook activity via an Integration Pipeline, you can pass values to the Notebook at runtime by tagging a dedicated cell in the Notebook as the Parameters Cell.

There is a small indication at the bottom right of the cell stating this is the parameters cell. There can only be one per notebook. 

Within this cell, you define variables that you want to pass the parameter values to at run time. You can provide default values to them for use when building and debugging the notebook in isolation. You reference these variables as any normal variable in the language you are working in.
In the pipeline where you are calling this notebook, add parameters to the notebook activity, using the exact same names as the variables defined in the notebook itself.
At runtime, Synapse will take the values sent to the pipeline, pass them down to the Notebook and override the variables with the runtime values.
Pretty straightforward up to this point. However, here you can see that adding more parameters results in Integration Pipeline changes.

Extensible Parameters using JSON

To solve this issue, we will update the value being passed to the parameter to be a string representation of a JSON Object or JSON Array. The contents of that parameter can be changed to add any number of additional attributes over time, without causing pipeline definition changes. The triple single quotes here are needed in order to escape single or double quotes used in the actual JSON. 
In order to use the contents of the variable in the notebook, you capture this string into a string variable first. From there you need to convert it into a JSON Typed Object in order to work with it interactively using the json.loads function from the JSON library in python. Other languages will have other functions – this is note covered in this article. 
Now you can extract the specific attributes from the variable to use as parameters as needed. If you are passing in a JSON Array as I am here, you can extract information from a single element in the array using index notation.
Here are some samples showing the types are of the data and elements in this array for reference. This will help inform you on how to interact with this new variable. 
If you pass in an array you need to loop over, you can use a regular python for loop to do so – if you are using python in your notebook that is.
To add this notebook to the pipeline, you can use the button top right to add it to an existing or a new integration pipeline.
The parameter config in the activity looks very similar – it is still a string variable, but you will pass in the JSON String instead of a scalar string value.
If you ever need additional parameters to be sent down to the notebook at runtime, alter the JSON String being sent to the single parameter, and update the Notebook code to utilize the additional attribute it will find in the JSON, without ever altering the Pipeline definition at all.

Benefits of this approach

  1. Avoid Integration Pipeline alteration when adding parameters to a notebook – cutting down on development time and reducing risk in introducing bugs.  
  2. The complexity of the Notebook can grow larger to perform more advanced tasks dynamically without worrying about the complexity introduced into the Integration Pipeline.  
  3. You can make the Integration Pipeline activities even more dynamic to call a different notebook at runtime depending on other runtime variables, not just a dedicated/hardcoded notebook. Since you can make each notebook receive the exact same parameter , you aren’t limited to calling a hardcoded Notebook in an activity. This makes the Integration Pipelines even more dynamic and metadata driven – increasing efficiency in development time.  

If you like what I do please consider supporting me on Ko-Fi

43 thoughts on “Parameterize Synapse Analytics Spark Notebooks Efficiently”

  1. Excellent post. I was checking constantly this blog and I am impressed! Extremely helpful info specially the last part 🙂 I care for such info much. I was looking for this certain information for a very long time. Thank you and good luck.

  2. I was suggested this blog by my cousin. I am now not sure whether this publish is written by him as no one else recognise such
    particular approximately my problem. You’re amazing!
    Thank you!

  3. when should i prune forsythia‘ve learn several good stuff here.
    Definitely value bookmarking for revisiting. I surprise how so much attempt you put to create the sort
    of wonderful informative site.

    best night vision for hunting
    mace pepper spray gun review
    bullshooter live
    how to make 8 cups of coffee
    how to work a bunn coffee maker
    anti glare night vision glasses
    keurig vue v500
    military surplus night vision goggles
    what’s the difference between mocha and latte
    vue keurig

  4. Hey! Quick question that’s entirely off topic. Do you know how to make your site mobile friendly? My blog looks weird when viewing from my apple iphone 13. I’m trying to find a template or plugin that might be able to correct this issue. If you have any recommendations, please share.

  5. Incredible blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your web site is magnificent, let alone the content!

  6. Great post! I was checking continuously this blog and I am impressed! Extremely useful info specially the last part 🙂 I care for such information a lot.

  7. Enjoyed studying this, very good stuff, thankyou . «To be positive To be mistaken at the top of one’s voice.» by Ambrose Bierce.

  8. Hey there great website! Does running a blog similar to this take a large amount of work?
    I have virtually no understanding of computer programming
    however I had been hoping to start my own blog soon. Anyway, should you have any suggestions or techniques for new blog
    owners please share. I know this is off subject nevertheless I simply wanted to ask.
    Many thanks!

    anti glare lenses cost
    welding in aviation
    gen 1 night vision binoculars
    diy dartboard
    john deere zero turn mower reviews
    bunn coffeemaker
    simplicity champion zero turn mower reviews
    juniper bonsai indoors
    lawn mower front wheels
    best electronic dart board

    My blog post … hoshizaki ice machines reset button

  9. Thank you for the auspicious writeup. It in fact was a amusement account it. Look advanced to more added agreeable from you! By the way, how can we communicate?

  10. Aw, this was an exceptionally nice post. Taking a few minutes and
    actual effort to make a really good article… but what can I say…
    I put things off a lot and never seem to get nearly anything done.

    Have a look at my page maple tree symbolism

  11. Great post! I was checking continuously this blog and I am impressed! Extremely useful info specially the last part 🙂 I care for such information a lot.

  12. folks that has been posted Im anxious . really did it for me is all the interaction among posters at such good info you have.

  13. Pretty! This was an extremely wonderful post.
    Thank you for providing these details.

    why does my bunn coffee maker overflow
    best night vision binoculars under $500
    bunn programmable coffee maker
    nautalis sleep system
    aws d17 1 certification
    troy bilt model 13an779g766
    manual lawn plug aerator
    john deere mower oil change
    grass catcher for craftsman riding mower
    craftsman mower decks for sale

    my web page:

  14. Hmm it appears like your website ate my first comment (it was super long) so I guess I’ll just sum it
    up what I wrote and say, I’m thoroughly enjoying your blog.
    I as well am an aspiring blog writer but
    I’m still new to everything. Do you have any points for
    inexperienced blog writers? I’d certainly appreciate it.

    where to buy bunn coffee pots
    mount dart board
    coffe maker with grinder
    coffee packs for percolators
    aws d17 1 certification
    cub cadet snowblower oil change
    bunn coffee pots on sale
    lavender star flower bulbs
    arachnid 800 dartboard
    bushnell falcon 10×50 wide angle binoculars

    Feel free to visit my web blog;

    1. Thank you for your kind words, really appreciate it!
      My best advise is to just start. Even if you are writing offline and not publishing, just start. Each small step is a step in the right direction.
      Try to write as if you are writing for one specific person that is reading it, yourself.

  15. You should be a part of a contest for one of the greatest sites on the internet.
    I am going to highly recommend this blog!

    best way to make your beard grow faster
    kyocera kd315gx lpb
    breville bes840xl the infuser espresso machine
    coffee pots that grind beans
    bunn coffeemaker
    john deere d125 lawn tractor reviews
    armasight vega
    anti glare prescription glasses
    <a href="hbunn stainless steel coffee potbunn stainless steel coffee pot 3 pot coffee maker

  16. Hello There. I discovered your blog using msn. That is a really well written article.
    I’ll be sure to bookmark it and return to learn more of your helpful information. Thank you for the post.

    I’ll certainly return.

    grinding coffee makers
    cleaning my coffee maker with vinegar
    night vision for hog hunting
    the best night vision binoculars
    amount of caffeine in instant coffee
    10 cup bunn coffee maker
    what is the best night vision binoculars
    how to make coffee in a large percolator
    how to clean a krups coffee maker
    dartboard height

  17. Excellent post. I used to be checking constantly
    this blog and I am inspired! Very useful info particularly the ultimate section 🙂 I take care of such information much.
    I was looking for this particular info for a very
    long time. Thanks and good luck.

  18. I am extremely impressed with your writing skills and also with the layout on your blog.
    Is this a paid theme or did you modify it yourself? Either
    way keep up the nice quality writing, it is rare to see a great blog like this one today.

  19. Hi! Do you know if they make any plugins to assist with SEO?
    I’m trying to get my blog to rank for some targeted keywords but I’m not
    seeing very good success. If you know of any please share.

  20. Hi there, You have done a great job. I’ll definitely digg it and personally
    recommend to my friends. I’m confident they’ll be benefited from this website.

  21. Hi, the whole thing is going sound here and ofcourse every one is sharing facts,
    that’s truly fine, keep up writing.

  22. If you would like to take a great deal from this piece of writing
    then you have to apply such strategies to your won web site.

    1. Hi !
      I simply used the Elementor plugin to build all my pages. It looks like they are mobile friendly enough for my taste.
      Try it out and see if that will be sufficient for your use case as well.

Comments are closed.