13th of May, 2021

Service

New Features

Parallel Tasks

We now support running tasks within a job in parallel. You can do this by using the new slider option on tasks to enable the parallel setting. When multiple tasks next to each other in sequence are set to parallel they will be sent to your agent(s) at the same time and run, you can even set all tasks in a job to run in parallel.

Parallel Scripts

Much like a data migration, you can now configure script tasks that source scripts from a folder or a GIT connection to run in parallel. Once you enable the parallel option, you can choose how many scripts to run in parallel, or opt to run them all in parallel. This only limited by the availability of processing power on the VM or container running the agent.

You can use the parallel option with the following task types when you source scripts from a folder or a GIT connection;

  • SQL Statement
  • OS Command
  • Python
  • PowerShell Core
  • Windows PowerShell
  • Spark SQL Statement

Improvements

  • We have rebuilt the Aconex driver to provide a wider support of the Aconex API.
  • We have also enabled the ability to perform incremental data loads on the Aconex and Microfocus ALM drivers.
  • We have added support to the Azure Shipyard task for singularity containers.

OS Command Task

We have made a number of changes to the OS command task to resolve issues with this task.

If you use OS command tasks in conjunction with task parameters, please review the below and check your task script prior to upgrading your agent as you may need to make adjustments.

  • Rather than passing the script itself directly as a parameter to cmd in Windows or bash in Linux, we now temporarily write the script to a .sh file in Linux and a .bat file in Windows under the /temp directory in the agent folder. This script is deleted once the script has finished running. This will resolve issues with multi-line OS command scripts that would not run depending on what types of commands were used in the script.
  • Task parameter values are now quoted and passed left to right in both cmd in Windows and bash in Linux. This will allow you to have variables with spaces in the value.
  • Task parameter values are trimmed for white space, if the parameter value contains nothing after being trimmed it will not be passed as an argument.
  • Task parameters can be accessed in descending order as %1, %2, %3, etc in Windows and as $1, $2, $3, etc in Linux.

Fixes

  • We have resolved an issue with data migrations using Azure Data Lake where columns exceed 8K of data size.
  • We have resolved an issue with persistent staging relating to column size changes of existing columns from the source data.
  • We have resolved an issue where a failure of a task depended upon by another task did not prevent the subsequent task to fail.
  • We have resolved an issue that sometimes caused tenant syncing to fail.

Agent

Minimum Version Number: 2021.05.13.1

  • To take advantage of the above changes, please update your agent to the latest version.