top of page
  • Writer's pictureCathy Basil-Jones

ForEach, Variables and Parallelism in Azure Data Factory (Part 2)

Updated: Oct 19, 2023

You'll remember last time we talked about a limitation with using ForEach activities in Azure Data Factory ("ADF"), setting variables and parallelism. Today we're going to work through the workaround. The JSON for both pipelines can be found at the bottom of this post.


The Set Up

Last time our pipeline consisted of an array variable, v_outer_array, a ForEach activity, nested If Condition activity as well as setting a child variable v_foreach followed by a grandchild variable v_ifcondition.


In this version, we split our pipeline into two pipelines - pipeline_parent and pipeline_child - with pipeline_parent calling pipeline_child within the ForEach activity. We replace the variable, v_foreach, with a parameter, p_foreach, which is passed into pipeline_child from the current item of v_outer_array.


Let's take a closer look.


Similar to last time, pipeline_parent consists of an array v_outer_array of two values - "a" and "b" and a ForEach activity.


Nested within the ForEach activity, there is an Execute Pipeline activity which calls pipeline pipeline_child. The current value of v_outer_array is passed into this pipeline as parameter p_foreach.


pipeline_child consists of an If Condition activity where we check whether p_foreach equates to "b".


Regardless of whether the If Condition activity is True or False, we set v_ifcondition to the current value of p_foreach.


The Results

As expected, pipeline_child is invoked twice, passing in both "a" and "b" as parameter p_foreach.


This results in v_ifcondition being set correctly to both "a" and "b".


Summary

In this post, we looked at a workaround to be able to use ForEach activities, set variables and parallelism in ADF. We were successfully able to set our variable by creating and invoking a child pipeline from our parent pipeline.

pipeline_parent
.txt
Download TXT • 2KB
pipeline_child
.txt
Download TXT • 2KB

Comments


bottom of page