Table of Contents
Introduction
VMware NSX Advanced Load Balancer (Avi) is a self-service Multi-Cloud App Services Platform that ensures consistent application delivery, with software load balancers, web application firewall (WAF), and container Ingress solutions for app's across data centers and clouds.
Avi can be installed on multiple clouds, but the process differs based on cloud resources and services. In this blog, I have presented a simplified way to install and configure VMware NSX Advanced Load Balancer (Avi) on AWS.
Preparation
Before we start with installation, it's good to make certain checks and perform configurations in AWS.
AWS credentials for Avi Setup
There are 2 ways to connect Avi with AWS -
- access credentials
- IAM roles
Though, access credentials for a user with sufficient privileges can be used to connect Avi with AWS, but it's highly recommended to configure IAM roles.
Setup IAM roles
To configure Avi to use AWS resources, we have to create 2 roles with corresponding policies in AWS.
- vmimport
- AviController-Refined-Role
Avi team have already prepared necessary roles and policies which can be readily applied via AWS CLI or console. These roles and policies can be found in https://github.com/avinetworks/devops.git project at subpath devops/aws/iam-policies
. Operator can also refer role install docs for more clarity on the roles.
Some operators would prefer using terraform for role creation, thus I have created the terraform scripts for implementing Avi roles on AWS which can be found at https://github.com/rajks24/avi-awsroles-terraform.git.
IAM roles can be easily applied using below AWS CLI commands:
vmimport role :
For AviController-Refined-Role role :
First, we would create the policies for the role.
Next step, would be to attach the policies to the AviController-Refined-Role
role.
NOTE: In the above commands, we need to replace the
projectId
to corresponding AWS Account.
To complete the role setup, we create the instance profile for AviController-Refined-Role, which will enable role to be attach with EC2 instance.
NSX Advanced Load Balancer supports deployment on AWS configured with multiple
AWS accounts utilizing the IAM AssumeRole functionality. Cross-Account Assume
Role provide access across AWS accounts to the AWS resources/API from the
respective accounts. While creating the AWS cloud-type, option of Use Cross-Account AssumeRole
is available on the NSX Advanced Load Balancer. The
Use Cross-Account AssumeRole feature can be enabled if the AWS cloud needs to
be created in an AWS account other than the one that hosts the Controller.
Detailed steps can be found
here.
Avi Sizing
Detailed sizing requirement for Avi can be referred here.
To summarize, we need following sizing for Avi controller :
- Controller: 8 vCPU cores, 24 GB RAM, and 128 GB of storage
- Service Engine: 1 vCPU cores, 2 GB RAM, and 15 GB of storage
Avi controller sizing on AWS
NSX Advanced Load Balancer recommends general purpose or compute/ memory optimized instances for running Avi Controllers.
Size | Instance Type |
---|---|
Small | m4.2xlarge |
Medium | C4.4xlarge , m4.4xlarge |
Large | C4.8xlarge, m4.10xlarge |
Burstable instances are not recommended for running Controller virtual machines. NSX Advanced Load Balancer Controller recommends SE with minimum memory of 2 GB, and 1 vCPU.
Typical HA deployment for Avi has three Controllers. The number of SEs required depends on number of applications being served and the configured level of redundancy.
Network Requirements
NSX Advanced Load Balancer Service Engine data interfaces can be assigned to multiple VRFs (Virtual Routing and Forwarding Context).
Ports and protocol required for Avi (v22.1.4) in a restricted environment can be found here.
In case, we are looking for ports related to a different version of Avi ( in future), we can refer the VMware global ports and protocol page here.
Avi Controller
Avi controller can be developed as a single EC2 instance. It can be configured with cloud setup to connect kubernetes cluster and SEs to provision virtual services.
NSX Advanced Load Balancer Service Engines handle all data plane operations within the NSX Advanced Load Balancer by receiving and executing instructions from the Controller. It performs load balancing and all client and server-facing network interactions. SE collects real-time telemetry data from application traffic flows.
Avi Controller install process
We can get the latest NSX Advanced Load Balancer AMI from AWS Marketplace.
AWS provides the manual launch (EC2 Console) process for the EC2 instance, where we can provide following information during the installation process.
- Select the AWS region for EC2
- Instance type can be m5.2xlarge ( for other compatible type, refer avi-sizing)
- Associate the instance with key-pair
- Pre-configured VPC and subnet for Avi EC2
- Auto assign public IP ( if Avi controller to be installed in public subnet).
- Security group that allows traffic through the firewall, to allow communication between the Controller and the Service Engines (SEs)
- SSH (22)
- http (80)
- https (443)
- custom-tcp (8443)
- udp (123)
- Storage 128gb or more
- Select IAM Instance profile - AviController-Refined-Role
With above configuration, Avi controller instance would start provisioning. It might take some time to get configured. Once the instance is running, we can access it's web interface at port 443 with the assigned Public/Private IPv4 address or Public IPv4 domain.
Initial password setup
Log in to the instance with SSH using user as admin
and assigned ec2 key-pair and use the sudo /opt/avi/scripts/initialize_admin_user.py
script to configure the admin password for the first time login to the Controller
Initial Avi setup
Access Avi GUI and provide the passphrase password ( It's used for Avi instance data backup setup which can be performed in later stage), DNS (optional) and save changes to move for the aws cloud setup in Avi.
AWS Cloud Setup
Operator can navigate in GUI to infrastructure section and create a new cloud of AWS
type and configure following details .
- AWS credentials ( Ensure to choose iam role option to use configured role).
- Select the AWS region
- Availability Zone & Service Engine Management Network ( we can select 1 or more AZ with subnets [private/public] which would be used to provision Service Engine instances)
- Select Use Encryption for SE S3 Bucket and Use Encryption for SE AMI/EBS volumes (Optional)
- DNS provider ( It's recommended to select Amazon Route 53 )
Some users might get an error while configuring AWS Credentials as Role. The error might look something like below. To resolve the error we need to verify that the AWS role has got all the policies attached.
Configure Avi license
VMware NSX Tanzu customers can subscribe for Avi license based on their requirements. Avi can be configured with license key based on their entitlements. Accordingly Avi would get configured with features and required core counts. These cores would be used by Service Engine and define the usage. The details for licensing can be found here.
Operator can configure Avi's license key for Enterprise , Basic or Essential tier by navigating to Avi console at Administration > Licensing section.
Avi with Enterprise Tier can be configured with 1 months expiry for evaluation license with 22 service cores for testing purpose. After 1 month Avi with Enterprise Tier can be degraded to Trial license with 2 service cores for testing purpose in non-prod environments.
Configure Service Engine Group
Operator can navigate to infrastructure > cloud resources and verify the default SE Group for configured cloud. We can either use the default SE Group created for the aws cloud in Avi or we can create a new SE group instance. All the options and fields are pre-populated and it's fine to start Avi SE Group with defaults. These fields can be changed later.
NSX Advanced Load Balancer SE groups support following HA modes:
Elastic HA: It provides fast recovery for individual virtual services following the failure of the SE. Depending on the mode, the virtual service is already running on multiple SEs or is quickly placed on another SE.
The following modes of cluster HA are supported:
- Active/Active
- N + M
Legacy HA: It emulates a 2-device hardware active/standby HA configuration operation. The active SE carries all the traffic for a virtual service placed on it. The other SE in the pair is the standby for the VS, carrying no traffic when the active SE is healthy.
For getting more details to configure SE in Elastic HA mode, we can refer this page.
Once SE Group is configured, revisit cloud section and update the Template Service Engine Group
option for the configured cloud in Avi.
NOTE: Default value for Service Engine Name Prefix
as Avi
should match to the value in S3 IAM policy. It's recommended to leave it unchanged.
The above config would kickoff cloud config in Avi and it would temporarily put the Avi SE AMI in S3 bucket.
Then, the AMI would be finally saved in AWS within EC2 AMI section.
NOTE: At any point, we can verify the events under Operations section, to check for any error or task performed by Avi for configurations.
Now, Avi controller is setup with the Service Engine and can be configured for L4 and L7 use cases.
We need to ensure Avi license is configured, else virtual machines for SE won't get provisioned after completing above steps.
Conclusion
With the above changes, NSX Advanced Load Balancer (Avi) is configured with license for aws cloud and configurations for SE engine. These configs are sufficient to start with setting up Kubernetes cluster or services to connect with Avi.
As the next step, we can configure a Kubernetes cluster (Amazon EKS) with Avi Kubernetes Operator (AKO) to connect with provisioned Avi controller and launch LoadBalancer service or ingress resource for a deployed application. Step by step process is discussed in next post ➡️ Deploying Kubernetes Operator for Avi on Amazon EKS