Data Collection

Jul 26, 2024

Billing Data

AWS stores billing data, the Cost and Usage Reports (CUR), for all accounts of an AWS Organization within the Organization’s management account (also sometimes referred to as a “master account”). If the client owns only one AWS account, for this purpose that account is the management account.

The CUR must be stored in an AWS S3 bucket before it can be retrieved. The Cloudsaver application creates an S3 bucket to hold this data. The bucket will be named “cloudsaver-xxxxxxxxxxxx-billing-files” where “xxxxxxxxxxxx” will be replaced with the client’s management account ID. Note, the provided permission policy limits the application’s access only to this S3 bucket.

Cloudsaver adds a 90-day lifecycle policy on this S3 bucket. This provides enough cushion to redownload previous billing files if necessary, while keeping the size and cost of the S3 bucket relatively small.

Cloudsaver also configures a CUR to send the billing files to the S3 bucket. The CUR configuration will publish a new report twice a day, once in the morning and once in the evening.

Tag Data

The Cloudsaver app retrieves environmental and operational data via AWS’ standard API. The permission policies attached to the Cloudsaver role allow the application to read tags on resources for usage within the app and when the client takes action within the app, to list, create and delete tags.

The permission policies also permit further read, list, and describe actions for certain resources, for example AWS EC2 instances and RDS databases. The Cloudsaver application uses those read actions to enrich tag data with the most-commonly required data points, for example the subnet ID of an AWS EC2 instance.

Tag Policy Data

The Cloudsaver application also enables more efficient management of tag policies in AWS Organizations. These can be used to standardize tags across resources in client Organization member accounts and for example, to programmatically enforce certain key casing or validate values when tags are created throughout the Organization’s member accounts.

The Cloudsaver permission policies allow the application to read tag policies within AWS Organizations and to create or modify tag policies when such action is taken by the client user within the application.

Resource and Log Data

The Cloudsaver application uses Amazon EventBridge to capture real time resource data changes and provide the most up-to-date visibility for the client. EventBridge notifies the Cloudsaver application (via API) whenever relevant resources data changes within the client’s environment. The application leverages this stream of event data to provide real-time synchronization between the client’s AWS environment and their view of resources within the application. As a baseline, the Cloudsaver application performs a daily resource data intake from AWS, against which EventBridge events are compared to reflect changes as they occur in the client’s environment.