Deploy Secure HDInsight Cluster

Following is an excerpt from HDInsight Security using Managed Identities Chapter from book Exam AZ-500 Study & Lab Guide Part 2: Microsoft Certified Azure Security Engineer Associate.

A financial Institution in Insurance sector wants to deploy HDInsight Cluster to do Big Data Analysis on its existing Customer base. This is being done to find favourable prospects within their customer base to whom they can sell additional financial products. Because of Regulatory compliance reasons the disks associated with HDInsight Cluster must be encrypted.

Note 1: To know more about HDInsight Cluster Architecture & User Assigned Managed Identity Refer to Chapter Topics.
Note 2: To know more about Azure Key Vault refer to Azure Key Vault Chapter.

Step 1: Create a user-assigned managed identity

In this step we will create user-assigned identity in Resource Group RGCloud. We will assign this identity to HDInsight Cluster in Step 4.

In Azure Portal click Create a resource> In search box enter user assigned managedidentity and press enter> User Assigned Manged identity pane opens. Click create> Create user assigned managed identity blade opens> Enter a name>Select RGCloud in Resource group> In location select East US 2> Click create.

2. Figure below shows dashboard of User Assigned Manged identity uamicloud.

Step 2: Allow user-assigned identity uamicloud to access Key Vault

In this Step we will allow user-assigned identity uamicloud (created in previous Step) to access Key Vault kvcloudd. Key Vault kvcloudd was created in Key Vault Chapter.

Go to Key Vault kvcloudd Dashboard and click Access Policies in left pane> Note the + Add access policy option in right pane.

2. Click + Add access policy option in right pane> Add access policy balde opens> In Key, Secret and Certificate permissions dropdown boxes Click select All> Click Select Principal>Select a Principal blade opens> In search box enter uamicloud> In result select uamicloud> Click Select> Click Add.

3. Back in kvcloudd dashboard click save in right pane. You can see the uamicloud added to access policies.

Step 3: Copy Symmetric Key sakey URL identifier

Symmetric Key sakey will be used for Disk Encryption in next Step where we will create HDInsight Cluster. Symmetric key sakey was created in Azure Key Vault Chapter.

1. Go to Key Vault kvcloudd Dashboard and click Keys in left pane.

2. Click the Key sakey in right pane>sakey version pane opens.Click the current version> Key version pane opens>Copy Key identifier.

Step 4: Create Secure HDInsight Cluster

In this exercise we will create HDInsight Cluster of type Hadoop in Virtual Network VNETCloud2. We will encrypt HDInsight Cluster disks with Symmetric key sakey. We will assign User assigned identity uamicloud to HDInsight Cluster.

VNETCloud2 was created In Virtual Network Chapter. Symmetric sakey was created in Azure Key Vault Chapter.

1. In Azure Portal Click Create a Resource> Analytics> Azure HDInsight> Create HDInsight Cluster blade Opens>Select Resource Group RGCloud> Enter a name> Select Region East US 2>Click Select cluster type> Select Cluster type blade opens. Select Hadoop> Enter Cluster Login Name> Enter Cluster password>Click Next: Storage.

2. In Storage Pane for Primary Storage type select Azure Storage> For Primary Storage Account select Automatically created (New)hdicloudhdistorage. The Storage Account name will be derived from name of the cluster. Please note that container will be automatically created> Rest Select all default values>Click Next: Security + Networking.

3. In Security + Networking pane note the Enterprise security package option. We are not enabling it here>In Minimum TLS version select 1.2> In Virtual Network select VNETCloud2>In Subnet select default subnet> In Disk Encryption Settings select Provide your own key from Key Vault> In Key URL enter the Key URL of sakey copied in previous step> In user assigned managed identity select uamicloud created in Step 1>Click Next: Configuration + pricing (Not Shown).

4. In Configuration + pricing pane I reduced the worker nodes from 4 to 2.

5. Click Review + create> System will validate the cluster configuration. After validation is succeeded click create (Not Shown). It will take around 35 minutes for Cluster creation.

6. Figure below shows dashboard of HDInsight cluster hdicloud. Note the Cluster URL https://hdicloud.azurehdinsight.net. Note the Disk Encryption Key/URL. This shows disks are encrypted.

To know more about Secure HDInsight Cluster and other topics in HDInsight Security using Managed Identities Chapter such as HDInsight Cluster Architecture, User Assigned Managed Identity and Security Options in HDInsight Cluster refer to book Exam AZ-500 Study & Lab Guide Part 2: Microsoft Certified Azure Security Engineer Associate.

The Book is now available on Amazon.