In my previous post, I have discussed about how to create vpc, subnets, internet gateway, nat gateway using AWS CDK(Python). In this post, we will be discussing about how to create EKS (k8s) cluster. This is requested by many of my followers, so I would like to write a blog to help the community.
Create a new stack called EKSStack and use existing vpc from other stack (vpc: ec2.Vpc)
In the above code, you can see that I’m using version 1.9, cluster will use private subnets and the IAM role that was created will be used. I used default_capacity = 0 since, I do not want EKS to create default node group. I will be creating node groups separately and will use SPOT instances for worker nodes in this case. This is all about control plane definition.
I’ve used multiple instance types for SPOT for different worker nodes. Disk size for the nodes to set to 50GB and two nodes will be created. you can adjust these parameters based on your requirement.
you need to update requirements.txt file to import modules for iam and eks. you can install these modules using pip install -r requirements.txt
aws-cdk.aws-iam
aws-cdk.aws-eks
now, let’s run cdk ls to see howmany stacks we can see. we should expect to see two stacks.
Now, let’s Synthesize an AWS CloudFormation template for the app, as follows.
(.venv) C:\Rama\MyProjects\mycdkproject>cdk synth
Successfully synthesized to C:\Rama\MyProjects\mycdkproject\cdk.out
Supply a stack id (mycdkproject, eks) to display its template.
Now, let’s deploy
cdk deploy eks --profile cdkprofile
if you have encountered an error as eks failed: Error: This stack uses assets, so the toolkit stack must be deployed to the environment (Run “cdk bootstrap aws://unknown-account/unknown-region”)
you can fix this by running cdk bootstrap aws://accountnumber/region
So far, i’ve discussed about how to configure vpc, subnets, natgateway, ec2 using terraform, in this post i’m going to discuss how to configure VPC, Subnets, Internet Gateway, NatGateway with AWS CDK using python as language. We don’t need to write any complex cloudformation scripts, we use AWS CDK to construct resources. For more information about AWS CDK, you may check here.
Pre-requisites:
You need to install aws cdk and python installed on your environment.
Once, you have aws cdk and python installed we can initialize new project. I’m using visual studio code in my example to create project, you can use any of your favorite IDE.
mkdir mycdkproject
cdk init --language python
The above will create a cdk structure with python as language. Activate your virtual environment source .env/bin/activate or python -m venv .venv depending on your OS (Linux/MaC/Windows). Once you initialized then the structure would look like this
We have cdk structure ready, let us start importing required modules (aws_ec2, aws_ssm, core)
edit the requirements.txt file (
aws-cdk.core aws-cdk.aws-ssm aws-cdk.aws-ec2) and install them using python install -r requirements.txt
This will ensure you have required modules to start with.
Let us modify the mycdkproject_stack.py.
from aws_cdk import (
aws_ec2 as ec2,
aws_ssm as ssm,
core
)
class MycdkprojectStack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# The code that defines your stack goes here
env_name = self.node.try_get_context("env")
self.vpc = ec2.Vpc(self, 'demovpc',
cidr = '192.168.50.0/24',
max_azs = 2,
enable_dns_hostnames = True,
enable_dns_support = True,
subnet_configuration=[
ec2.SubnetConfiguration(
name = 'Public-Subent',
subnet_type = ec2.SubnetType.PUBLIC,
cidr_mask = 26
),
ec2.SubnetConfiguration(
name = 'Private-Subnet',
subnet_type = ec2.SubnetType.PRIVATE,
cidr_mask = 26
)
],
nat_gateways = 1,
)
priv_subnets = [subnet.subnet_id for subnet in self.vpc.private_subnets]
count = 1
for psub in priv_subnets:
ssm.StringParameter(self, 'private-subnet-'+ str(count),
string_value = psub,
parameter_name = '/'+env_name+'/private-subnet-'+str(count)
)
count += 1
The above script will create vpc, public, private subnets and nategateway.
Call this stack into your app.py as below
#!/usr/bin/env python3
from aws_cdk import core
from mycdkproject.mycdkproject_stack import MycdkprojectStack
app = core.App()
MycdkprojectStack(app, "mycdkproject")
app.synth()
let us run cdk ls to see if there any list of stacks
cdk ls
Synthesize an AWS CloudFormation template for the app, as follows.
cdk synth
If your app contained more than one stack, you’d need to specify which stack(s) to synthesize. But since it only contains one, the Toolkit knows you must mean that one.
The cdk synth command executes your app, which causes the resources defined in it to be translated to an AWS CloudFormation template. The displayed output of cdk synth is a YAML-format template; our app’s output is shown below. The template is also saved in the cdk.out directory in JSON format.
Oh, well how simple is that? Very minimal code and you don’t need to write large cloud formation templates. This is just a sample example, i’m going to upload full project in my github repository (https://github.com/sankar276/awscdkpython)
Please note that this is just a demonstration of how to create EC2 and ALB using terraform and i’ve not created certificate for alb example here. You can create ssl cert and attach it to alb.
Now, let us do terraform plan
terraform plan
provider.aws.region
The region where AWS operations will take place. Examples
are us-east-1, us-west-2, etc.
Enter a value: eu-west-1
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.template_file.s3_ec2_policy: Refreshing state...
data.aws_availability_zones.azs: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_eip.nat will be created
+ resource "aws_eip" "nat" {
+ allocation_id = (known after apply)
+ association_id = (known after apply)
+ customer_owned_ip = (known after apply)
+ domain = (known after apply)
+ id = (known after apply)
+ instance = (known after apply)
+ network_border_group = (known after apply)
+ network_interface = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ public_ipv4_pool = (known after apply)
+ vpc = true
}
# aws_iam_instance_profile.ec2_profile will be created
+ resource "aws_iam_instance_profile" "ec2_profile" {
+ arn = (known after apply)
+ create_date = (known after apply)
+ id = (known after apply)
+ name = "ec2_profile"
+ path = "/"
+ role = "ec2_role"
+ unique_id = (known after apply)
}
# aws_iam_role.ec2_role will be created
+ resource "aws_iam_role" "ec2_role" {
+ arn = (known after apply)
+ assume_role_policy = jsonencode(
{
+ Statement = [
+ {
+ Action = "sts:AssumeRole"
+ Effect = "Allow"
+ Principal = {
+ Service = "ec2.amazonaws.com"
}
},
]
+ Version = "2012-10-17"
}
)
+ create_date = (known after apply)
+ force_detach_policies = false
+ id = (known after apply)
+ max_session_duration = 3600
+ name = "ec2_role"
+ path = "/"
+ unique_id = (known after apply)
}
# aws_iam_role_policy.ec2_policy will be created
+ resource "aws_iam_role_policy" "ec2_policy" {
+ id = (known after apply)
+ name = "ec2_policy"
+ policy = jsonencode(
{
+ Statement = [
+ {
+ Action = [
+ "s3:ListStorageLensConfigurations",
+ "s3:ListAllMyBuckets",
+ "s3:ListJobs",
+ "ec2:*",
]
+ Effect = "Allow"
+ Resource = "*"
+ Sid = "Stmt202102171510"
},
+ {
+ Action = [
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:ListBucketMultipartUploads",
+ "s3:ListBucketVersions",
+ "s3:ListBucket",
+ "s3:ListMultipartUploadParts",
]
+ Effect = "Allow"
+ Resource = [
+ "arn:aws:s3:::terraform-demo-202102171103/*",
+ "arn:aws:s3:::terraform-demo-202102171103/*/*",
]
+ Sid = "Stmt202102171530"
},
]
+ Version = "2012-10-17"
}
)
+ role = (known after apply)
}
# aws_instance.web[0] will be created
+ resource "aws_instance" "web" {
+ ami = "ami-0fc970315c2d38f01"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ iam_instance_profile = "ec2_profile"
+ id = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t3.medium"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "appserver"
}
+ tenancy = (known after apply)
+ user_data = "4cf0e018e9af98d1b7ebab63b0c032d4a207b7ef"
+ volume_tags = (known after apply)
+ vpc_security_group_ids = (known after apply)
+ ebs_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ snapshot_id = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
+ ephemeral_block_device {
+ device_name = (known after apply)
+ no_device = (known after apply)
+ virtual_name = (known after apply)
}
+ metadata_options {
+ http_endpoint = (known after apply)
+ http_put_response_hop_limit = (known after apply)
+ http_tokens = (known after apply)
}
+ network_interface {
+ delete_on_termination = (known after apply)
+ device_index = (known after apply)
+ network_interface_id = (known after apply)
}
+ root_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
}
# aws_instance.web[1] will be created
+ resource "aws_instance" "web" {
+ ami = "ami-0fc970315c2d38f01"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ iam_instance_profile = "ec2_profile"
+ id = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t3.medium"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "appserver"
}
+ tenancy = (known after apply)
+ user_data = "4cf0e018e9af98d1b7ebab63b0c032d4a207b7ef"
+ volume_tags = (known after apply)
+ vpc_security_group_ids = (known after apply)
+ ebs_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ snapshot_id = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
+ ephemeral_block_device {
+ device_name = (known after apply)
+ no_device = (known after apply)
+ virtual_name = (known after apply)
}
+ metadata_options {
+ http_endpoint = (known after apply)
+ http_put_response_hop_limit = (known after apply)
+ http_tokens = (known after apply)
}
+ network_interface {
+ delete_on_termination = (known after apply)
+ device_index = (known after apply)
+ network_interface_id = (known after apply)
}
+ root_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
}
# aws_internet_gateway.igw will be created
+ resource "aws_internet_gateway" "igw" {
+ arn = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "mydemo-igw"
}
+ vpc_id = (known after apply)
}
# aws_lb.myapp will be created
+ resource "aws_lb" "myapp" {
+ arn = (known after apply)
+ arn_suffix = (known after apply)
+ dns_name = (known after apply)
+ drop_invalid_header_fields = false
+ enable_deletion_protection = true
+ enable_http2 = true
+ id = (known after apply)
+ idle_timeout = 60
+ internal = false
+ ip_address_type = (known after apply)
+ load_balancer_type = "application"
+ name = "myapp-lb-tf"
+ security_groups = (known after apply)
+ subnets = (known after apply)
+ tags = {
+ "Environment" = "demo"
}
+ vpc_id = (known after apply)
+ zone_id = (known after apply)
+ subnet_mapping {
+ allocation_id = (known after apply)
+ outpost_id = (known after apply)
+ private_ipv4_address = (known after apply)
+ subnet_id = (known after apply)
}
}
# aws_lb_listener.myapp will be created
+ resource "aws_lb_listener" "myapp" {
+ arn = (known after apply)
+ id = (known after apply)
+ load_balancer_arn = (known after apply)
+ port = 80
+ protocol = "HTTP"
+ ssl_policy = (known after apply)
+ default_action {
+ order = (known after apply)
+ target_group_arn = (known after apply)
+ type = "forward"
}
}
# aws_lb_target_group.myapp will be created
+ resource "aws_lb_target_group" "myapp" {
+ arn = (known after apply)
+ arn_suffix = (known after apply)
+ deregistration_delay = 300
+ id = (known after apply)
+ lambda_multi_value_headers_enabled = false
+ load_balancing_algorithm_type = (known after apply)
+ name = "myapp-lb-tg"
+ port = 80
+ protocol = "HTTP"
+ proxy_protocol_v2 = false
+ slow_start = 0
+ target_type = "instance"
+ vpc_id = (known after apply)
+ health_check {
+ enabled = (known after apply)
+ healthy_threshold = (known after apply)
+ interval = (known after apply)
+ matcher = (known after apply)
+ path = (known after apply)
+ port = (known after apply)
+ protocol = (known after apply)
+ timeout = (known after apply)
+ unhealthy_threshold = (known after apply)
}
+ stickiness {
+ cookie_duration = (known after apply)
+ enabled = (known after apply)
+ type = (known after apply)
}
}
# aws_lb_target_group_attachment.myapp[0] will be created
+ resource "aws_lb_target_group_attachment" "myapp" {
+ id = (known after apply)
+ port = 80
+ target_group_arn = (known after apply)
+ target_id = (known after apply)
}
# aws_lb_target_group_attachment.myapp[1] will be created
+ resource "aws_lb_target_group_attachment" "myapp" {
+ id = (known after apply)
+ port = 80
+ target_group_arn = (known after apply)
+ target_id = (known after apply)
}
# aws_nat_gateway.ngw will be created
+ resource "aws_nat_gateway" "ngw" {
+ allocation_id = (known after apply)
+ id = (known after apply)
+ network_interface_id = (known after apply)
+ private_ip = (known after apply)
+ public_ip = (known after apply)
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "NatGateway"
}
}
# aws_route_table.privatert will be created
+ resource "aws_route_table" "privatert" {
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = [
+ {
+ cidr_block = "0.0.0.0/0"
+ egress_only_gateway_id = ""
+ gateway_id = ""
+ instance_id = ""
+ ipv6_cidr_block = ""
+ local_gateway_id = ""
+ nat_gateway_id = (known after apply)
+ network_interface_id = ""
+ transit_gateway_id = ""
+ vpc_endpoint_id = ""
+ vpc_peering_connection_id = ""
},
]
+ tags = {
+ "Name" = "mydemoprivatert"
}
+ vpc_id = (known after apply)
}
# aws_route_table.publicrt will be created
+ resource "aws_route_table" "publicrt" {
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = [
+ {
+ cidr_block = "0.0.0.0/0"
+ egress_only_gateway_id = ""
+ gateway_id = (known after apply)
+ instance_id = ""
+ ipv6_cidr_block = ""
+ local_gateway_id = ""
+ nat_gateway_id = ""
+ network_interface_id = ""
+ transit_gateway_id = ""
+ vpc_endpoint_id = ""
+ vpc_peering_connection_id = ""
},
]
+ tags = {
+ "Name" = "mydemopublicrt"
}
+ vpc_id = (known after apply)
}
# aws_route_table_association.private_subnet_association[0] will be created
+ resource "aws_route_table_association" "private_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.private_subnet_association[1] will be created
+ resource "aws_route_table_association" "private_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.pub_subnet_association[0] will be created
+ resource "aws_route_table_association" "pub_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.pub_subnet_association[1] will be created
+ resource "aws_route_table_association" "pub_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_s3_bucket.my_bucket will be created
+ resource "aws_s3_bucket" "my_bucket" {
+ acceleration_status = (known after apply)
+ acl = "private"
+ arn = (known after apply)
+ bucket = "terraform-demo-202102171103"
+ bucket_domain_name = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = false
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ tags = {
+ "Environment" = "demo"
+ "Name" = "My bucket"
}
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
+ versioning {
+ enabled = (known after apply)
+ mfa_delete = (known after apply)
}
}
# aws_security_group.alb_sg will be created
+ resource "aws_security_group" "alb_sg" {
+ arn = (known after apply)
+ description = "Allow inbound traffic for web applicaiton on ec2"
+ egress = (known after apply)
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = "alb web port"
+ from_port = 80
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 80
},
]
+ name = "alb_sg"
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Name" = "alb_sg"
}
+ vpc_id = (known after apply)
}
# aws_security_group.ec2_sg will be created
+ resource "aws_security_group" "ec2_sg" {
+ arn = (known after apply)
+ description = "Allow inbound traffic for web applicaiton on ec2"
+ egress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = false
+ to_port = 0
},
]
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = [
+ "192.168.50.0/24",
]
+ description = "ssh port"
+ from_port = 22
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 22
},
+ {
+ cidr_blocks = [
+ "192.168.50.0/24",
]
+ description = "web port"
+ from_port = 80
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 80
},
]
+ name = "ec2_sg"
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Name" = "ec2_sg"
}
+ vpc_id = (known after apply)
}
# aws_subnet.private[0] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1a"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.128/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PrivateSubnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.private[1] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1b"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.192/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PrivateSubnet-2"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[0] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1a"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.0/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PublicSubnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[1] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1b"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.64/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PublicSubnet-2"
}
+ vpc_id = (known after apply)
}
# aws_vpc.mydemodemo will be created
+ resource "aws_vpc" "mydemodemo" {
+ arn = (known after apply)
+ assign_generated_ipv6_cidr_block = false
+ cidr_block = "192.168.50.0/24"
+ default_network_acl_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_group_id = (known after apply)
+ dhcp_options_id = (known after apply)
+ enable_classiclink = (known after apply)
+ enable_classiclink_dns_support = (known after apply)
+ enable_dns_hostnames = true
+ enable_dns_support = true
+ id = (known after apply)
+ instance_tenancy = "default"
+ ipv6_association_id = (known after apply)
+ ipv6_cidr_block = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Environment" = "demo"
+ "Name" = "mydemodemo"
}
}
Plan: 27 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ nat_gateway_ip = (known after apply)
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
If you can see that plan is creating ec2, ec2 roles, ec2 security groups, attaching ec2_role to ec2 instance, creating target groups, alb, listerners. Let us apply terraform
That’s it. if you see that the ec2 and alb’s are created. you can customize the script based on your requirement.
Continuation to the previous article, I’m going to demonstrate how to configure vpc, public subnet, internet gateway, private subnet, NAT Gateway. I’ve modified the scripts to simplify and easy to manage.
I’ve created two terraform workspace as demo so that the terraform will create all resources on demo workspace. This will help us to isolate different environments when you are working on multiple environments such as dev,qa,uat,prod. terraform state files will be created based on the workspace environment.
I’ve used cidrsubent function to create subnets without have to provide them manually. I’ve also used slice and length functions to extract consecutive elements within a list of availability zones and calculate the length to loop through each availability zone to create subnet id’s. For more details, you may refer terraform documentation.
The above terraform file creates, elastic ip address, nat gateway, private subnets, route tables, route table associations.
Please note that i’ve only created two public and two private subnets in this example. If you do want to create subnets for each availability zones then you can modify
count = length(slice(local.az_names)).
Also modify cidr function according to your cidr range. I’m using /24 in this example, if you would like to use larger range /16 cidr then you can modify the function as below
cidrsubnet(var.vpc_cidr, 8, count.index). you can also pass newbits(8) as parameter to generate subnets according to your requirement.
Let us test this and apply.
terraform plan
terraform plan
provider.aws.region
The region where AWS operations will take place. Examples
are us-east-1, us-west-2, etc.
Enter a value: eu-west-1
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_availability_zones.azs: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_eip.nat will be created
+ resource "aws_eip" "nat" {
+ allocation_id = (known after apply)
+ association_id = (known after apply)
+ customer_owned_ip = (known after apply)
+ domain = (known after apply)
+ id = (known after apply)
+ instance = (known after apply)
+ network_border_group = (known after apply)
+ network_interface = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ public_ipv4_pool = (known after apply)
+ vpc = true
}
# aws_internet_gateway.igw will be created
+ resource "aws_internet_gateway" "igw" {
+ arn = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "mydemo-igw"
}
+ vpc_id = (known after apply)
}
# aws_nat_gateway.ngw will be created
+ resource "aws_nat_gateway" "ngw" {
+ allocation_id = (known after apply)
+ id = (known after apply)
+ network_interface_id = (known after apply)
+ private_ip = (known after apply)
+ public_ip = (known after apply)
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "NatGateway"
}
}
# aws_route_table.privatert will be created
+ resource "aws_route_table" "privatert" {
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = [
+ {
+ cidr_block = "0.0.0.0/0"
+ egress_only_gateway_id = ""
+ gateway_id = ""
+ instance_id = ""
+ ipv6_cidr_block = ""
+ local_gateway_id = ""
+ nat_gateway_id = (known after apply)
+ network_interface_id = ""
+ transit_gateway_id = ""
+ vpc_endpoint_id = ""
+ vpc_peering_connection_id = ""
},
]
+ tags = {
+ "Name" = "mydemoprivatert"
}
+ vpc_id = (known after apply)
}
# aws_route_table.publicrt will be created
+ resource "aws_route_table" "publicrt" {
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = [
+ {
+ cidr_block = "0.0.0.0/0"
+ egress_only_gateway_id = ""
+ gateway_id = (known after apply)
+ instance_id = ""
+ ipv6_cidr_block = ""
+ local_gateway_id = ""
+ nat_gateway_id = ""
+ network_interface_id = ""
+ transit_gateway_id = ""
+ vpc_endpoint_id = ""
+ vpc_peering_connection_id = ""
},
]
+ tags = {
+ "Name" = "mydemopublicrt"
}
+ vpc_id = (known after apply)
}
# aws_route_table_association.private_subnet_association[0] will be created
+ resource "aws_route_table_association" "private_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.private_subnet_association[1] will be created
+ resource "aws_route_table_association" "private_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.pub_subnet_association[0] will be created
+ resource "aws_route_table_association" "pub_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.pub_subnet_association[1] will be created
+ resource "aws_route_table_association" "pub_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_subnet.private[0] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1a"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.128/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PrivateSubnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.private[1] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1b"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.192/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PrivateSubnet-2"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[0] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1a"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.0/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PublicSubnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[1] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1b"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.64/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PublicSubnet-2"
}
+ vpc_id = (known after apply)
}
# aws_vpc.mydemodemo will be created
+ resource "aws_vpc" "mydemodemo" {
+ arn = (known after apply)
+ assign_generated_ipv6_cidr_block = false
+ cidr_block = "192.168.50.0/24"
+ default_network_acl_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_group_id = (known after apply)
+ dhcp_options_id = (known after apply)
+ enable_classiclink = (known after apply)
+ enable_classiclink_dns_support = (known after apply)
+ enable_dns_hostnames = true
+ enable_dns_support = true
+ id = (known after apply)
+ instance_tenancy = "default"
+ ipv6_association_id = (known after apply)
+ ipv6_cidr_block = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Environment" = "demo"
+ "Name" = "mydemodemo"
}
}
Plan: 14 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ nat_gateway_ip = (known after apply)
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
If you look the plan, it creates vpc, internet gateway, route tables, public subnets, elastic ip, nat gateway, private subnets, route table association.
Plan looks good, let us apply.
terraform apply
terraform apply
provider.aws.region
The region where AWS operations will take place. Examples
are us-east-1, us-west-2, etc.
Enter a value: eu-west-1
data.aws_availability_zones.azs: Refreshing state...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_eip.nat will be created
+ resource "aws_eip" "nat" {
+ allocation_id = (known after apply)
+ association_id = (known after apply)
+ customer_owned_ip = (known after apply)
+ domain = (known after apply)
+ id = (known after apply)
+ instance = (known after apply)
+ network_border_group = (known after apply)
+ network_interface = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ public_ipv4_pool = (known after apply)
+ vpc = true
}
# aws_internet_gateway.igw will be created
+ resource "aws_internet_gateway" "igw" {
+ arn = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "mydemo-igw"
}
+ vpc_id = (known after apply)
}
# aws_nat_gateway.ngw will be created
+ resource "aws_nat_gateway" "ngw" {
+ allocation_id = (known after apply)
+ id = (known after apply)
+ network_interface_id = (known after apply)
+ private_ip = (known after apply)
+ public_ip = (known after apply)
+ subnet_id = (known after apply)
+ tags = {
+ "Name" = "NatGateway"
}
}
# aws_route_table.privatert will be created
+ resource "aws_route_table" "privatert" {
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = [
+ {
+ cidr_block = "0.0.0.0/0"
+ egress_only_gateway_id = ""
+ gateway_id = ""
+ instance_id = ""
+ ipv6_cidr_block = ""
+ local_gateway_id = ""
+ nat_gateway_id = (known after apply)
+ network_interface_id = ""
+ transit_gateway_id = ""
+ vpc_endpoint_id = ""
+ vpc_peering_connection_id = ""
},
]
+ tags = {
+ "Name" = "mydemoprivatert"
}
+ vpc_id = (known after apply)
}
# aws_route_table.publicrt will be created
+ resource "aws_route_table" "publicrt" {
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = [
+ {
+ cidr_block = "0.0.0.0/0"
+ egress_only_gateway_id = ""
+ gateway_id = (known after apply)
+ instance_id = ""
+ ipv6_cidr_block = ""
+ local_gateway_id = ""
+ nat_gateway_id = ""
+ network_interface_id = ""
+ transit_gateway_id = ""
+ vpc_endpoint_id = ""
+ vpc_peering_connection_id = ""
},
]
+ tags = {
+ "Name" = "mydemopublicrt"
}
+ vpc_id = (known after apply)
}
# aws_route_table_association.private_subnet_association[0] will be created
+ resource "aws_route_table_association" "private_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.private_subnet_association[1] will be created
+ resource "aws_route_table_association" "private_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.pub_subnet_association[0] will be created
+ resource "aws_route_table_association" "pub_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.pub_subnet_association[1] will be created
+ resource "aws_route_table_association" "pub_subnet_association" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_subnet.private[0] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1a"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.128/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PrivateSubnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.private[1] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1b"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.192/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PrivateSubnet-2"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[0] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1a"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.0/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PublicSubnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[1] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "eu-west-1b"
+ availability_zone_id = (known after apply)
+ cidr_block = "192.168.50.64/26"
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "PublicSubnet-2"
}
+ vpc_id = (known after apply)
}
# aws_vpc.mydemodemo will be created
+ resource "aws_vpc" "mydemodemo" {
+ arn = (known after apply)
+ assign_generated_ipv6_cidr_block = false
+ cidr_block = "192.168.50.0/24"
+ default_network_acl_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_group_id = (known after apply)
+ dhcp_options_id = (known after apply)
+ enable_classiclink = (known after apply)
+ enable_classiclink_dns_support = (known after apply)
+ enable_dns_hostnames = true
+ enable_dns_support = true
+ id = (known after apply)
+ instance_tenancy = "default"
+ ipv6_association_id = (known after apply)
+ ipv6_cidr_block = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Environment" = "demo"
+ "Name" = "mydemodemo"
}
}
Plan: 14 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ nat_gateway_ip = (known after apply)
Do you want to perform these actions in workspace "demo"?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_eip.nat: Creating...
aws_vpc.mydemodemo: Creating...
aws_eip.nat: Creation complete after 0s [id=eipalloc-0575af76798f68266]
aws_vpc.mydemodemo: Creation complete after 1s [id=vpc-0d96547a55dd596f6]
aws_subnet.public[1]: Creating...
aws_subnet.private[0]: Creating...
aws_internet_gateway.igw: Creating...
aws_subnet.private[1]: Creating...
aws_subnet.public[0]: Creating...
aws_internet_gateway.igw: Creation complete after 1s [id=igw-06d576b66aec53673]
aws_route_table.publicrt: Creating...
aws_subnet.private[1]: Creation complete after 1s [id=subnet-0ab88aee1b33b8483]
aws_subnet.private[0]: Creation complete after 1s [id=subnet-0ce8e8f1440ffd973]
aws_nat_gateway.ngw: Creating...
aws_subnet.public[0]: Creation complete after 1s [id=subnet-000a4de59eecee2a3]
aws_subnet.public[1]: Creation complete after 1s [id=subnet-0d53209d4e1208cd9]
aws_route_table.publicrt: Creation complete after 0s [id=rtb-046e80cb1d2e37801]
aws_route_table_association.pub_subnet_association[1]: Creating...
aws_route_table_association.pub_subnet_association[0]: Creating...
aws_route_table_association.pub_subnet_association[0]: Creation complete after 0s [id=rtbassoc-0eb5f538c92bf0a9b]
aws_route_table_association.pub_subnet_association[1]: Creation complete after 0s [id=rtbassoc-088d81f0ac032d6db]
aws_nat_gateway.ngw: Still creating... [10s elapsed]
aws_nat_gateway.ngw: Still creating... [20s elapsed]
aws_nat_gateway.ngw: Still creating... [30s elapsed]
aws_nat_gateway.ngw: Still creating... [40s elapsed]
aws_nat_gateway.ngw: Still creating... [50s elapsed]
aws_nat_gateway.ngw: Still creating... [1m0s elapsed]
aws_nat_gateway.ngw: Still creating... [1m10s elapsed]
aws_nat_gateway.ngw: Still creating... [1m20s elapsed]
aws_nat_gateway.ngw: Still creating... [1m30s elapsed]
aws_nat_gateway.ngw: Still creating... [1m40s elapsed]
aws_nat_gateway.ngw: Creation complete after 1m44s [id=nat-094789de3dcb5d9a9]
aws_route_table.privatert: Creating...
aws_route_table.privatert: Creation complete after 0s [id=rtb-09c75e35733f44b0d]
aws_route_table_association.private_subnet_association[1]: Creating...
aws_route_table_association.private_subnet_association[0]: Creating...
aws_route_table_association.private_subnet_association[1]: Creation complete after 1s [id=rtbassoc-0d97f7027cdf93594]
aws_route_table_association.private_subnet_association[0]: Creation complete after 1s [id=rtbassoc-00dd10f8578438faa]
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
Outputs:
nat_gateway_ip = xx.xx.xxx.xxx
That’s it. Stay tuned for more updates on this series (Terraform, Ansible, AWS CDK)
I used element function to get the list of availability zones and create them respectively. For more details about element function you may refer here
Now, let’s run terraform plan
terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_availability_zones.AZs: Refreshing state…
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
create
Terraform will perform the following actions:
# aws_internet_gateway.mydemodemo-igw will be created
resource "aws_internet_gateway" "mydemodemo-igw" {
arn = (known after apply)
id = (known after apply)
owner_id = (known after apply)
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-private-a will be created
resource "aws_subnet" "mydemodemo-private-a" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1c"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.64/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-private-a"
}
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-private-b will be created
resource "aws_subnet" "mydemodemo-private-b" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1a"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.192/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-private-b"
}
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-public-a will be created
resource "aws_subnet" "mydemodemo-public-a" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1a"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.0/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-public-a"
}
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-public-b will be created
resource "aws_subnet" "mydemodemo-public-b" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1b"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.128/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-public-b"
}
vpc_id = (known after apply)
}
aws_vpc.mydemodemo will be created
resource "aws_vpc" "mydemodemo" {
arn = (known after apply)
assign_generated_ipv6_cidr_block = false
cidr_block = "192.168.50.0/24"
default_network_acl_id = (known after apply)
default_route_table_id = (known after apply)
default_security_group_id = (known after apply)
dhcp_options_id = (known after apply)
enable_classiclink = (known after apply)
enable_classiclink_dns_support = (known after apply)
enable_dns_hostnames = true
enable_dns_support = true
id = (known after apply)
instance_tenancy = "default"
ipv6_association_id = (known after apply)
ipv6_cidr_block = (known after apply)
main_route_table_id = (known after apply)
owner_id = (known after apply)
tags = { "Name" = "mydemodemo"
}
}
Plan: 6 to add, 0 to change, 0 to destroy.
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
Terraform plan looks good. you can see that the resources are going to be created without any errors.
Here is the full network.tf file
Create VPC
resource "aws_vpc" "mydemodemo" {
provider = aws.region
cidr_block = "192.168.50.0/24"
instance_tenancy = "default"
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "mydemodemo"
}
}
Create internet gateways
resource "aws_internet_gateway" "mydemodemo-igw" {
provider = aws.region
vpc_id = aws_vpc.mydemodemo.id
}
Get all Availability Zones in the current vpc mydemodemo
data "aws_availability_zones" "AZs" {
provider = aws.region
state = "available"
}
Create public subnet # 1 in eu-west-1 region
resource "aws_subnet" "mydemodemo-public-a" {
provider = aws.region
availability_zone = element(data.aws_availability_zones.AZs.names, 0)
vpc_id = aws_vpc.mydemodemo.id
cidr_block = "192.168.50.0/26"
tags = {
Name = "mydemodemo-public-a"
}
}
Create public subnet # 2 in eu-west-1 region
resource "aws_subnet" "mydemodemo-public-b" {
provider = aws.region
availability_zone = element(data.aws_availability_zones.AZs.names, 1)
vpc_id = aws_vpc.mydemodemo.id
cidr_block = "192.168.50.128/26"
tags = {
Name = "mydemodemo-public-b"
}
}
Create private subnet #1 in eu-west-1 region
resource "aws_subnet" "mydemodemo-private-a" {
provider = aws.region
availability_zone = element(data.aws_availability_zones.AZs.names, 2)
vpc_id = aws_vpc.mydemodemo.id
cidr_block = "192.168.50.64/26"
tags = {
Name = "mydemodemo-private-a"
}
}
resource "aws_subnet" "mydemodemo-private-b" {
provider = aws.region
availability_zone = element(data.aws_availability_zones.AZs.names, 3)
vpc_id = aws_vpc.mydemodemo.id
cidr_block = "192.168.50.192/26"
tags = {
Name = "mydemodemo-private-b"
}
}
Let us apply network.tf
terraform apply
terraform apply
data.aws_availability_zones.AZs: Refreshing state…
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
create
Terraform will perform the following actions:
# aws_internet_gateway.mydemodemo-igw will be created
resource "aws_internet_gateway" "mydemodemo-igw" {
arn = (known after apply)
id = (known after apply)
owner_id = (known after apply)
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-private-a will be created
resource "aws_subnet" "mydemodemo-private-a" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1c"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.64/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-private-a"
}
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-private-b will be created
resource "aws_subnet" "mydemodemo-private-b" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1a"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.192/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-private-b"
}
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-public-a will be created
resource "aws_subnet" "mydemodemo-public-a" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1a"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.0/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-public-a"
}
vpc_id = (known after apply)
}
aws_subnet.mydemodemo-public-b will be created
resource "aws_subnet" "mydemodemo-public-b" {
arn = (known after apply)
assign_ipv6_address_on_creation = false
availability_zone = "eu-west-1b"
availability_zone_id = (known after apply)
cidr_block = "192.168.50.128/26"
id = (known after apply)
ipv6_cidr_block_association_id = (known after apply)
map_public_ip_on_launch = false
owner_id = (known after apply)
tags = { "Name" = "mydemodemo-public-b"
}
vpc_id = (known after apply)
}
aws_vpc.mydemodemo will be created
resource "aws_vpc" "mydemodemo" {
arn = (known after apply)
assign_generated_ipv6_cidr_block = false
cidr_block = "192.168.50.0/24"
default_network_acl_id = (known after apply)
default_route_table_id = (known after apply)
default_security_group_id = (known after apply)
dhcp_options_id = (known after apply)
enable_classiclink = (known after apply)
enable_classiclink_dns_support = (known after apply)
enable_dns_hostnames = true
enable_dns_support = true
id = (known after apply)
instance_tenancy = "default"
ipv6_association_id = (known after apply)
ipv6_cidr_block = (known after apply)
main_route_table_id = (known after apply)
owner_id = (known after apply)
tags = { "Name" = "mydemodemo"
}
}
Plan: 6 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_vpc.mydemodemo: Creating…
aws_vpc.mydemodemo: Creation complete after 1s [id=vpc-00566cc74b0162d72]
aws_subnet.mydemodemo-public-b: Creating…
aws_subnet.mydemodemo-private-b: Creating…
aws_subnet.mydemodemo-private-a: Creating…
aws_internet_gateway.mydemodemo-igw: Creating…
aws_subnet.mydemodemo-public-a: Creating…
aws_internet_gateway.mydemodemo-igw: Creation complete after 0s [id=igw-054372249d1864798]
aws_subnet.mydemodemo-private-b: Creation complete after 0s [id=subnet-0148fb04d52e33bca]
aws_subnet.mydemodemo-public-b: Creation complete after 0s [id=subnet-0392daa47a4dfec77]
aws_subnet.mydemodemo-private-a: Creation complete after 0s [id=subnet-0a2c8222c5acfc2c0]
aws_subnet.mydemodemo-public-a: Creation complete after 0s [id=subnet-014010a55e3cb336a]
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
You can see that 6 resources are created. Wait for few minutes to see them appear on aws console.
Here it is, I can see that vpc, network, internet gateway are created.
That’s it. you can see that we have created network (VPC, subnets, Internet gateway). I will be creating EC2, RDS in the upcoming articles. Stay tuned.
In this blog post, I would like to present how to create terraform state file as s3. I will be writing series of blog posts about terraform, ansible automation, How to use AWS CDK to automate infrastructure.
To begin with, Let us start about how to configure terraform state file as Amazon S3.
Before you configure terraform state, you need to make sure terraform install. For more details about how to install terraform you may refer here
Create s3 bucket (You can do this by using aws console or aws cli)
I’ve used eu-west-1 region for creating bucket if you are using default region us-east-1 then you do not need to mention region, Locationconstraint.
The below script is used to create a database manual snapshots on all regions in an AWS account. RDS support automatic snapshots which are very useful for point in time recovery but it does not protect you from accidental deletion of RDS instances.
Here are the reasons why we need manual snapshots
Accidental RDS Instance Deletion
Cross region backup
Long term retention
import boto3
import time
import sys
def lambda_handler(event, context):
ec2Regions = boto3.client('ec2')
awsRegions = ec2Regions.describe_regions()['Regions']
for region in awsRegions:
rds = boto3.client('rds',region_name=region['RegionName'])
try:
current_date = time.strftime("%Y-%m-%d-%H-%M-%S")
print (current_date)
print (region)
response = rds.describe_db_instances()
try:
for rdsinstance in response['DBInstances']:
if (rdsinstance['DBInstanceStatus']=='available'):
try:
shotIdentifier = rdsinstance['DBInstanceIdentifier'].replace('_','-') + current_date
rds.create_db_snapshot(DBInstanceIdentifier = rdsinstance['DBInstanceIdentifier'],DBSnapshotIdentifier = shotIdentifier)
except Exception as e:
print ('Error::%s'%e)
else:
print ('Instance NOT available Instance State is %s and Engine is %s'%(rdsinstance['DBInstanceStatus'],rdsinstance['Engine']))
except Exception as e:
print ('Error::%s'%e)
except Exception as e:
print (e)
#lambda_handler(None,None)
One of the requirement was to generate csv file for set of queries from RDS PostgreSQL and upload the csv file to s3 bucket for power bi reporting. Powerbi connects to s3 url and generate report. There is no gateway to connect to PostgreSQL instance from power-bi, hence we need to have a mechanism to upload the data to s3 so that powerbi can import it and generate reports. How can you generate csv file and upload to s3 bucket ?
There are multiple ways we can achieve this, one is to use ssm command send over as shell script and use copy command for postgreSQL to generate csv file and push it to s3. Another approach is use pandas module and dataframe to convert the data to csv and push it to s3.
Both the examples are as below. I retreived username and password from parameter store. For more information about parameter store you may refer here.
import boto3
import time
import sys
import os
ec2 = boto3.client('ec2')
ssm_client = boto3.client('ssm')
def lambda_handler(event, context):
def execute_ssm_command(client, commands, instance_ids):
resp = client.send_command(
DocumentName="AWS-RunShellScript",
Parameters={'commands': commands},
InstanceIds=instance_ids,
)
nodename = os.environ['region']+'.'+os.environ['env']+'.'+os.environ['app']+'.'+os.environ['company']+'.'+os.environ['role']
print os.environ['date']
print os.environ['code']
uname = ssm_client.get_parameter(Name='dbusername', WithDecryption=True)
username = uname['Parameter']['Value']
pwd = ssm_client.get_parameter(Name='dbpassword', WithDecryption=True)
password = pwd['Parameter']['Value']
response=ec2.describe_instances()
for reservation in response["Reservations"]:
for instance in reservation["Instances"]:
if(instance.get("Tags")):
for tag in instance['Tags']:
if(tag.get("Key")):
if (tag['Key'] in 'Name'):
if (tag['Value'] in nodename):
print tag['Value']
commands = ['if [ -z "${HOME:-}" ]; then export HOME="$(cd ~ && pwd)"; fi','sudo yum install mail -y', 'sudo yum install postgresql96 -y', '#!/bin/bash ', 'PGPASSWORD='+str(password)+' psql -h postgres -U '+str(username)+' -d dbname -c "\copy (select * from report where date= '+os.environ['date']+' and code= '+os.environ['code']+') to stdout csv header">/$HOME/s3Reportqa.csv', 'aws s3 cp /$HOME/s3Report.csv s3://'+os.environ['bucketpath']+'/', 'printf "Hi All, csv file has been generated successfully. " | mailx -vvv -s "report" -r "noreply@ramasankarmolleti.com" -S smtp="smtp" "sankar276@gmail.com"']
temp = instance["InstanceId"]
instance_ids = [temp]
print os.environ['region']+'.'+os.environ['env']+'.'+os.environ['app']+'.'+os.environ['company']+'.'+os.environ['role']
print instance_ids
execute_ssm_command(ssm_client, commands, instance_ids)
Parameters:
Second Method (Using Pandas module)
import psycopg2
import request
import boto3
import pandas as pd
from pandas import Series, DataFrame
import csv
conn_string = "host='dbinstancename' dbname='databasename' user='username' password='password'"
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
print ("Connected")
tablename = 'report'
date = '2019-07-11'
code = 'RAMA'
#cursor.execute("SELECT * FROM " + tablename +" Where date ="+ date + " and code = "+code+";")
cursor.execute("SELECT * FROM " + tablename +" limit 100;")
myresult = cursor.fetchall()
item_list = []
for i in myresult:
item = {'col1':i[0],
'col2':i[1],
'col3' :i[2],
'col4' :i[3],
'col5' :i[4],
'col6' :i[5],
'col7' :i[6],
'col8' :i[7],
'col9' :i[8]}
item_list.append(item)
concat = str(i[0]) + str(',') + str(i[1]) + str(',') + str(i[2]) + str(',') + str(i[3]) + str(',') + str(i[4]) + str(',') + str(i[5]) + str(',') + str(i[6]) + str(',') + str(i[7]) + str(',') + str(i[8])
# print (concat)
df = pd.DataFrame(data=item_list,columns=['col1','col2','col3','col4','col5','col6','col7','col8','col9'])
df.head(30)
print (df.head(40))
# importing the result to csv begins
df.to_csv('rama.csv')
print('csv generated')
# to push the datafram results to s3, we can use boto3 s3 resource as below
s3_resource = boto3.resource('s3')
s3_resource.Object(bucket, 'df.csv').put(Body=csv_buffer.getvalue())
You can schedule the lambda using cloudwatch events every 5 minutes to update the data in s3.
Ramasankar Molleti Blog 