Azure Batch

Overview

Warning

The Tower support for Azure Batch is currently in beta. Any feedback and suggestions are welcome.

Disclaimer

This guide assumes you have an existing Azure Account. Sign up for a free Azure account here.

There are two ways to create a Compute Environment for Azure Batch with Tower.

1. Tower Forge for Azure Batch automatically creates Azure Batch resources in your Azure account.

2. Tower Launch allows you to create a compute environment using existing Azure Batch resources.

If you don't yet have an Azure Batch environment fully set-up, following the Tower Forge guide is suggested. If you have been provided with an Azure Batch queue from your account administrator or if you have set up Azure Batch previously, follow the Tower Launch guide.

Forge

Warning

Follow these instructions if you have not pre-configured an Azure Batch environment. Note that this will create resources in your Azure account that you may be charged for by Azure.

Resource group

To create the necessary Azure Batch and Azure Storage accounts, we must first create a Resource Group in the region of your choice.

When you open this link you'll notice the Create new resource group dialog, as shown below.

1. Add the name for the resource group (for e.g. towerrg).

2. Select the preferred region for this resource group.

3. Click Review and Create to proceed to the review screen.

4. Click Create to create the resources.

Storage account

The next step is to create the necessary Azure Storage. When you open this link you'll notice the "Create a storage account" dialog, as shown below.

1. Add the name for the storage account (for e.g. towerrgstorage).

2. Select the preferred region for this resource group.

3. Click Review and Create to proceed to the review screen.

4. Click Create to create the Azure Storage account.

5. Next, create a Blob container within this storage account by navigating to your new storage account and clicking on Container as shown below.

6. Create a new Blob container by clicking on the + Container option and a new container dialog with open, choose a suitable name (e.g. towerrgstorage-container) as shown below.

7. Once the new Blob container is created, navigate to the Access Keys section of the storage account (e.g. towerrgstorage).

8. Store the access keys for the newly created Azure Storage account as shown below.

Batch account

The next step is to create the necessary Azure Storage. When you open this link you'll notice the "Create a batch account" dialog, as shown below.

1. Add the name for the storage account (for e.g. towerrgbatch).

2. Select the preferred region for this resource group.

3. Click Review and Create to proceed to the review screen.

4. Click Create to create the Azure Batch account.

Congratulations!

You have completed the Azure environment setup for Tower.

Forge compute environment

Tower Forge automates the configuration of an Azure Batch compute environment and queues required for the deployment of Nextflow pipelines.

Once the Azure resource setup is done, we can add a new Azure Batch environment in the Tower UI. To create a new compute environment, follow these steps:

1. In the navigation bar on the upper right, choose your account name then choose Compute environments and select New Environment.

2. Enter a descriptive name for this environment, for example Azure Batch (east-us), and select Azure Batch as the target platform.

3. Add new credentials by selecting the + button. Choose a name, e.g. tower credentials and add the Access key and Secret key. These are the keys we saved in the previous steps when creating the Azure resources.

Multiple credentials

You can create multiple credentials in your Tower environment.

4. Select a Region, for example eastus (East US), and in the Pipeline work directory enter the Azure blob container we created in the previous section e.g: az://towerrgstorage-container/work.

Warning

The blob container should be in the same Region as selected above.

5. Select the Config mode as Batch Forge and, optionally, add the default VM type depending on your quota limits. The default VM type is Standard_D4_v3.

6. Next, specify the maximum number of VMs you'd like to deploy in the VMs count field.

7. Enable the Autoscale option, if you'd like to automatically scale up (VMs count) and down (0 VMs) based on the number of tasks.

8. Enable the Dispose resources options, if you'd like Tower to automatically delete the deployed Pool once the workflow is complete.

Advanced options

9. Optionally, specify the Jobs cleanup policy to delete the jobs once the workflows execution is completed.

10. Optionally, specify the duration of the SAS token generated by Nextflow.

11. Finally, click on Create to finalize the compute environment setup. It will take approximately 20 seconds for all the resources to be created and then you will be ready to launch pipelines.

Amazing!

You now have everything you need to begin deploying massively scalable pipelines.

Jump to the documentation section for Launching Pipelines.

Launch

This section is for users with a pre-configured Azure environment. You will need an Azure Batch account, Azure Storage account already set up. In order to add a new compute environment in the Tower UI for existing Azure resources, follow these steps:

1. In the navigation bar on the upper right, choose your account name then choose Compute environments and select New Environment.

2. Enter a descriptive name for this environment, for example, Azure Batch (east-us) and select Azure Batch as the target platform.

3. Add new credentials by selecting the + button. Choose a name, e.g. tower credentials and add the Access key and Secret key. These are the keys we saved in the previous steps when creating the Azure resources.

Multiple credentials

You can create multiple credentials in your Tower environment.

4. Select a Region, for example eastus (East US), and in the Pipeline work directory enter the Azure blob container we created in the previous section e.g: az://towerrgstorage-container/work.

Warning

The blob container should be in the same Region as selected above.

5. Select the Config mode as Manual and add the name of the Azure Batch pool, provided to you by your Azure administrator, in the Compute Pool name section.

Advanced options

6. Optionally, specify the Jobs cleanup policy to delete the jobs once the workflows execution is completed.

7. Optionally, specify the duration of the SAS token generated by Nextflow.

8. Finally, click on Create to finalize the compute environment setup. It will take approximately 20 seconds for all the resources to be created and then you will be ready to launch pipelines.

Amazing!

You now have everything you need to begin deploying massively scalable pipelines.

Jump to the documentation section for Launching Pipelines.