boto, the esteemed Python SDK for the AWS API, is being retired in favor of boto3, which has been deemed “stable and recommended for general use.” There are at least two big enhancements in boto3:
Interfaces to AWS are driven automatically by JSON service descriptions rather than hand-coded. As a result, the features provided by the SDK are both more current and more portable since SDKs in many languages can use the same JSON descriptions.
Resources, collections and related high-level interfaces offer a more pythonic and object-oriented experience for interacting with AWS.
All of Boto3’s resource and client classes are generated at runtime. This means that you cannot directly inherit and then extend the functionality of these classes because they do not exist until the program actually starts running. However it is still possible to extend the functionality of classes through Boto3’s event system.
The documentation goes on to give some trivial examples, but it still wasn’t immediately obvious to me, a relative Python neophyte, how to use this in my own development. I recently had a project where the ability to extend boto3 would simplify my code, and I’d like to share what I did.
Many of my clients have complex applications with several AWS VPCs, external networks, and a lot of moving parts with many teams managing the various systems. I am often hired to help with the design and the security of these environments, and in that role I end up doing a lot of vulnerability assessment work.
Assessments require a lot of data collection about the network architecture, security group structure, permissions, and other settings in the environment. Previously I spent a lot of time clicking through the console and making notes, a tedious process that was both time consuming and error prone.
I set about coding up a tool to collect as much of the data as possible automatically, and also to draw some conclusions about the security of the system based on the data. I learned how simple it is to extend boto3 using the event system.
An Example: Security Group Rules
The ec2.SecurityGroup class in boto3 contain an IpPermissions attribute that represents the group’s inbound rules. The attribute is a list of rules in JSON format, looking something like this:
I wanted to analyze the rules, and converting this JSON blob to a proper Python object would allow me to work with individual rules more easily.
To do this, I extend boto3 with two classes: SecurityGroupRules, used to add an attribute called rules to SecurityGroup, and then SecurityGroupRule (note singular vs plural) to represent a single rule. The rules attribute can be created when the object is initialized. Something like this:
The setter method extracts the fields from the ip_permissions JSON blob and creates a SecurityGroupRule object for every rule.
SecurityGroupRule looks like:
Great, so how do we use these classes? Enter the event system magic: register the SecurityGroupRules class to be added when boto3 instantiates a SecurityGroup.
Whenever a SecurityGroup object is created using the handle returned from get_ec2_handle(), SecurityGroupRules will also be instantiated. As a trivial example:
security_groups is a list of SecurityGroup objects, each of which has a rules attribute that is a list of SecurityGroupRule objects. It’s now trivial to list all rules in all groups:
We can also easily search the rules for insecure configurations, which is more detail than I planned for this post. So what’s considered an insecure configuration? Well that’s also not really the point of the post, but since you’ve asked my opinion…
All ports to the zero network (0.0.0.0/0) (obvious)
Any ports to the zero network should at least be noted
Access to all or many ports for a large CIDR range, where large is my somewhat arbitrary definition - /24 or larger.
The idea is that even internal to a VPC, security groups should be as restrictive as possible.
The boto3 event system is a novel approach to allow more pythonic interactions with AWS resources. This example showed how to essentially subclass ec2.SecurityGroup to allow better analysis of rules. I also subclass ec2.Instance to extract additional instance properties which aren’t immediately available, like IAM policies and instance userdata.
Please hassle me about something in the comments or on Twitter if you want. Thanks for reading.