28:11
Day-7 | Live AWS Project using SHELL SCRIPTING for DevOps | AWS DevOps project| #devops #aws #2023
Abhishek.Veeramalla
Overview
This video demonstrates how to create a shell script to track AWS resource usage, focusing on EC2 instances, S3 buckets, Lambda functions, and IAM users. It explains the importance of cost management in cloud environments and how such scripts help identify underutilized or unused resources. The tutorial covers using AWS CLI commands, enhancing script readability with echo statements and `set -x` for debugging, and simplifying output using `jq` for JSON parsing. Finally, it touches upon automating script execution with cron jobs for daily reporting.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Organizations move to the cloud primarily for reduced maintenance overhead and cost-effectiveness.
- Cloud environments operate on a pay-as-you-go model, meaning you only pay for resources you actively use.
- A key responsibility for DevOps engineers is to monitor resource usage to ensure cost efficiency.
- Unused resources, like EBS volumes not attached to EC2 instances, can incur unnecessary costs.
Understanding the core benefits of cloud adoption, especially cost savings, highlights the critical role of resource monitoring in a DevOps function.
A developer creating 100 EC2 instances or EBS volumes that are left unused, leading to unnecessary AWS charges.
- A shell script can be used to gather information about AWS resource usage, offering an alternative to Python scripts or Lambda functions.
- The script will track common AWS resources: EC2 instances, S3 buckets, Lambda functions, and IAM users.
- AWS CLI commands like `aws s3 ls`, `aws ec2 describe-instances`, `aws lambda list-functions`, and `aws iam list-users` are used to fetch data.
- Adding comments and echo statements improves script readability and helps in debugging.
This chapter introduces the practical application of shell scripting combined with AWS CLI for a fundamental DevOps task: resource inventory.
Using the command `aws ec2 describe-instances` to retrieve detailed information about all running EC2 instances in an AWS account.
- The `set -x` command enables debug mode, printing each command as it's executed along with its output.
- Using `echo` statements before commands clarifies what information is being retrieved (e.g., 'Printing list of S3 buckets').
- The `jq` tool is essential for parsing JSON output from AWS CLI commands, allowing extraction of specific data like instance IDs.
- By piping the AWS CLI output to `jq`, the script can provide a cleaner, more focused report, extracting only necessary details like instance IDs.
Learning to refine script output and utilize debugging tools like `set -x` and `jq` is crucial for creating efficient, maintainable, and understandable automation scripts.
Using `aws ec2 describe-instances | jq '.Reservations[].Instances[].InstanceId'` to extract only the Instance IDs from the verbose JSON output of the `describe-instances` command.
- Cron jobs are used to schedule the automatic execution of scripts at specific times or intervals.
- This automates the process of generating daily resource usage reports without manual intervention.
- Integrating the shell script with a cron job ensures that reports are generated consistently, even if the engineer is unavailable.
- The script's output can be redirected to a file for easy review or further processing.
Automation via cron jobs is a cornerstone of DevOps, ensuring routine tasks are performed reliably and efficiently, freeing up engineers for more complex work.
Scheduling the `aws_resource_tracker.sh` script to run every day at 6 PM using a cron job to automatically generate a resource usage report.
Key takeaways
- Cloud adoption offers significant benefits in manageability and cost, but requires diligent resource monitoring.
- Shell scripting combined with AWS CLI is a powerful and accessible method for automating cloud resource management tasks.
- Effective debugging techniques, such as `set -x`, are vital for troubleshooting scripts.
- Tools like `jq` are indispensable for processing and extracting specific data from JSON outputs in cloud automation.
- Cron jobs are essential for scheduling and automating routine tasks like daily resource reporting.
- Creating practical projects like resource trackers can be valuable additions to a DevOps engineer's resume.
Key terms
Cloud InfrastructureMaintenance OverheadCost-EffectivenessPay-as-you-goDevOps EngineerAWS CLIShell ScriptingBashCron JobJQJSON ParserEC2 InstancesS3 BucketsLambda FunctionsIAM Users
Test your understanding
- What are the two primary reasons organizations migrate to cloud infrastructure, and how does resource monitoring relate to them?
- How can a shell script help a DevOps engineer manage cloud costs effectively?
- Explain the purpose of the `set -x` command in shell scripting and how it aids in debugging.
- What is `jq`, and why is it useful when working with AWS CLI commands in a shell script?
- How can cron jobs be used to automate the daily generation of AWS resource usage reports?