Let's start with a little memory refresher… AWS Direct Connect is a service that enables you to establish a dedicated network connection from your on-premises data centers to AWS. A key component of this service is the use of Border Gateway Protocol (BGP) communities, which play a vital role in network routing and traffic management.
I had many conversation lately with multiple customers who wanted to limit their scope of advertisements when it comes to Direct Connect with Public Virtual Interface (Public VIF) and/or how to perform routing engineering having multiple Direct Connect locations, so here I am writing a short quick guide about these functionalities. We are going to talk about BGP Communities, how to use them, how to manipulate routing to/from AWS/on-prem.
Here the Lab diagram for today topic:
Table of Content:
What is a Virtual Interface (VIF)?
First of all, just because I mentioned the Virtual Interfaces, I think it would be good to provide the reader with a brief intro about what they are and how they differ. AWS Direct Connect offers three types of Virtual Interfaces (VIFs): Public VIF, Private VIF, and Transit VIF, each serving distinct use cases and offering different capabilities.
Public VIF: A Public Virtual Interface allows access to AWS public services such as Amazon S3, Amazon DynamoDB, and other AWS services with public endpoints. When you use a Public VIF, you advertise your public IP prefixes to AWS and receive public IP prefixes for AWS services. This setup is typically used when you need to reach AWS services directly without routing through the internet, providing a more secure and constant/lower-latency connection to public AWS endpoints.
Private VIF: A Private Virtual Interface connects directly to your Virtual Private Cloud (VPC) resources over a private IP space. This type of VIF is ideal for accessing EC2 instances, RDS databases, and other resources within your VPC. Private VIFs enable private, secure communication between your on-premises data center and your AWS environment, bypassing the public internet entirely, which is crucial for sensitive or high-availability applications.
Transit VIF: A Transit Virtual Interface connects to an AWS Transit Gateway, which allows you to interconnect multiple VPCs and on-premises networks via a single interface. This setup is particularly useful for complex network architectures where multiple VPCs need to communicate with each other or with on-premises networks through a “centralized gateway”. The Transit VIF simplifies network management and reduces the number of required interfaces, promoting a hub-and-spoke model for routing. Additionally providing, through the transit gateway, network segmentation capabilities.
Each VIF type is designed to optimize different networking needs, from direct access to public AWS services, private VPC connectivity, to complex multi-VPC and on-premises network integration. I will not dive deep into more details as this post is not intended to go through Direct Connect specifics. Let’s start talking about BGP Communities…
Introduction to BGP Communities
BGP communities are tags that you can assign to your IP prefixes. These tags help in controlling routing policies and making routing decisions. In general BGP, these communities influence how routes are propagated, preferred, or filtered across different networks. They provide a mechanism for grouping prefixes to apply specific routing policies easily.
BGP Communities in AWS Direct Connect
In the context of AWS Direct Connect, BGP communities are used to manage the traffic flow between your on-premises network and AWS over multiple Direct Connect locations. Additionally, AWS provides specific community tags that help in controlling the scope and preference of routes advertised/learned from or to on-prem and AWS.
Manipulating Traffic in Multiple DX Locations (Private VIF/Transit VIF)
When you have multiple Direct Connect (DX) locations, you can use BGP community tags to influence routing preferences. AWS supports various BGP community tags:
7224:7100: LOW preference
7224:7200: MEDIUM preference
7224:7300: HIGH preference
By assigning these tags to your prefixes, you can control which DX connection AWS prefers for incoming and outgoing traffic.
Traffic Management from AWS to On-Premis
Using Private VIF or Transit VIF
For traffic from AWS to your on-premises network using Private Virtual Interfaces (Private VIF) or Transit Virtual Interfaces (Transit VIF), you can manipulate routing paths using BGP attributes such as BGP Communities, AS_PATH, and MED (Multi-Exit Discriminator). Here’s how you can achieve primary vs. secondary path manipulation:
BGP Communities and Local Preference: Set a higher preference (e.g., 7224:7300) on the primary connection and a lower one (e.g., 7224:7100) on the secondary connection to prefer the primary path.
AS_PATH Prepending: Increase the AS_PATH length on the secondary connection to make it less preferred compared to the primary connection.
MED Values: Set lower MED values on the primary path if local preference and AS_PATH are the same.
Usually, you would want to use BGP Communities when you have multiple locations and AS_Path Prepending within the same location in case you have multiple links.
These techniques ensure that AWS prefers your primary Direct Connect location for most traffic while falling back to secondary connections if the primary one fails.
Using Public VIF - Inbound Routing Policies
When dealing with public virtual interfaces (Public VIF), AWS provides specific BGP community tags for controlling the scope of prefix advertisements:
7224:9100: Local AWS Region
7224:9200: All AWS Regions on the same continent
7224:9300: Global (all public AWS Regions)
By applying these tags to your advertised prefixes, you can control the propagation scope within the AWS network. Additionally, you can filter prefixes received from AWS to ensure only the necessary routes are accepted. AWS also tags all its advertised routes with the NO_EXPORT community, meaning these routes should not be advertised outside your network.
Using Public VIF - Outbound Routing Policies
AWS Direct Connect allows you to specify which prefixes are advertised from AWS to your network by using BGP communities and local preference.
7224:8100: For routes originating from the same AWS Region as the Direct Connect location
7224:8200: For routes from the same continent
NO_TAG: For routes originating from other continents
Additionally, to manipulate the routing of outbound traffic, you can use these communities in conjunction with local preference settings. For example, assigning a higher local preference value to prefixes from a preferred Direct Connect location ensures that traffic flows through that location by default. This is valid if you have a single device on-prem. If you want to manipulate the routing preferences in case of multiple locations, AWS Public VIF supports “longest prefix match” preferring the advertisement with the most specific prefix for example:
192.168.0.0/24 - Preferred
192.168.0.0/16 - NOT Preferred
NO_EXPORT BGP Community
The NO_EXPORT BGP community tag indicates that the tagged route should not be advertised beyond the local AS (Autonomous System) boundary. This is useful in preventing route leaks and ensuring that specific routes are contained within your network.
Practical Example Implementation (Private + Transit VIF)
Consider a scenario where you have two Direct Connect locations (DX1 and DX2) and you want to set up an active/passive configuration:
Configure BGP Communities:
For DX1 (primary): Apply 7224:7300 to set a high preference.
For DX2 (secondary): Apply 7224:7100 to set a low preference.
AS_PATH Prepending:
On DX2, prepend the AS_PATH to make it less preferred.
MED Configuration:
Set a lower MED value on DX1 to ensure it’s preferred when all other attributes are equal.
Route Filters:
On your on-premises BGP routers, filter received AWS routes to ensure only the necessary prefixes are accepted, applying the NO_EXPORT tag as needed.
CISCO config example (single service config):
! Define the BGP neighbor and basic BGP configurationrouter bgp 65000 bgp log-neighbor-changes! Configure first AWS Direct Connect link (primary) neighbor 192.0.2.1 remote-as 7224 neighbor 192.0.2.1 description AWS Direct Connect Primary neighbor 192.0.2.1 password 7 <Your_BGP_Password> neighbor 192.0.2.1 route-map SET_HIGH_PREFERENCE out neighbor 192.0.2.1 send-community both! Configure second AWS Direct Connect link (secondary) neighbor 198.51.100.1 remote-as 7224 neighbor 198.51.100.1 description AWS Direct Connect Secondary neighbor 198.51.100.1 password 7 <Your_BGP_Password> neighbor 198.51.100.1 route-map SET_LOW_PREFERENCE out neighbor 198.51.100.1 send-community both! Define address family for IPv4 address-family ipv4 neighbor 192.0.2.1 activate neighbor 198.51.100.1 activate exit-address-family! Define prefix list for advertised routesip prefix-list BGP-ADVERTISED-PREFIXES seq 5 permit 192.168.0.0/16! Define route-map to set high preference for primary linkroute-map SET_HIGH_PREFERENCE permit 10 match ip address prefix-list BGP-ADVERTISED-PREFIXES set community 7224:7300! Define route-map to set low preference for secondary linkroute-map SET_LOW_PREFERENCE permit 10 match ip address prefix-list BGP-ADVERTISED-PREFIXES set community 7224:7100! Apply route-maps to BGP neighborsrouter bgp 65000 address-family ipv4 neighbor 192.0.2.1 route-map SET_HIGH_PREFERENCE out neighbor 198.51.100.1 route-map SET_LOW_PREFERENCE out exit-address-familyExplanation
BGP Neighbor Configuration:
192.0.2.1 is configured as the primary AWS Direct Connect link.
198.51.100.1 is configured as the secondary AWS Direct Connect link.
Route-Maps:
SET_HIGH_PREFERENCE: This route-map sets the community 7224:7300 for routes advertised to the primary link, giving it higher precedence.
SET_LOW_PREFERENCE: This route-map sets the community 7224:7100 for routes advertised to the secondary link, giving it lower precedence.
Prefix List:
BGP-ADVERTISED-PREFIXES: This prefix list specifies which routes are advertised to AWS. Here it permits the prefix 192.168.0.0/16.
Address Family Activation:
Both neighbors are activated under the IPv4 address family.
Applying Route-Maps:
The route-maps SET_HIGH_PREFERENCE and SET_LOW_PREFERENCE are applied to the respective BGP neighbors to set the desired preferences for the primary and secondary links.
This simple configuration ensures that the primary link (192.0.2.1) is preferred over the secondary link (198.51.100.1) by assigning a higher preference community to it. Adjust IP addresses and AS numbers as per your actual configuration needs. Please note that this is strictly a test environment so apply proper judgement when configuring production environments.
Conclusion
BGP communities in AWS Direct Connect provide a powerful tool for controlling and optimizing your network traffic. By understanding and effectively using these communities, you can achieve greater control over routing decisions, ensuring optimal performance and reliability for your connections to AWS. As usual, thank you for your time, hope this was useful, see you soon!