↓Skip to main content
IT Blog IT Blog
  1. All Posts/

DevOps-Bash-tools

Tools Shell

Hari Sekhon – DevOps Bash Tools


























git.io/bash-tools
800+ DevOps Shell Scripts and Advanced Bash environment.
Fast, Advanced Systems Engineering, Automation, APIs, shorter CLIs, etc.
Heavily used in many GitHub repos, dozens of DockerHub builds (Dockerfiles) and 600+ CI builds.

Summary

  • Scripts for many popular DevOps technologies, see Inventory below for more details
  • Advanced configs for common tools like Git, vim, screen, tmux, PostgreSQL psql etc…
  • CI configs for most major Continuous Integration products (see CI builds page)
  • CI scripts for a drop-in framework of standard checks to run in all CI builds, CI detection, accounting for installation differences across CI environments, root vs user, virtualenvs etc.
  • API scripts auto-handling authentication, tokens and other details to quickly query popular APIs with a few keystrokes just supplying the /path/endpoint
  • Advanced Bash environment – .bashrc + .bash.d/*.sh – aliases, functions, colouring, dynamic Git & shell behaviour enhancements, automatic pathing for installations and major languages like Python, Perl, Ruby, NodeJS, Golang across Linux distributions and Mac. See .bash.d/README.md
  • Installs the best systems packages –
    AWS CLI,
    Azure CLI,
    GCloud SDK,
    Digital Ocean CLI,
    Terraform,
    Terragrunt,
    GitHub CLI,
    Kubernetes
    kubectl &
    kustomize,
    Helm,
    eksctl,
    Docker-Compose,
    jq
    and many others… extensive package lists for servers and desktops for most major Linux distributions package managers and Mac

    • setup/ – contains even more scripts to download and install software, JDBC connectors, Mac OS X settings etc.
  • Utility Libraries used by many hundreds of scripts and builds across repos:
    • .bash.d/ – interactive library
    • lib/ – scripting and CI library
  • SQL Scripts – 100+ scripts for PostgreSQL, MySQL, AWS Athena + CloudTrail, Google BigQuery
  • Templates – templates for common programming languages and build configs
  • Kubernetes Configs – Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

See Also: similar DevOps repos in other languages
Hari Sekhon
Cloud & Big Data Contractor, United Kingdom
(ex-Cloudera, former Hortonworks Consultant)

(you’re welcome to connect with me on LinkedIn)

Quick Setup

To bootstrap, install packages and link in to your shell profile to inherit all configs, do:

curl -L https://git.io/bash-bootstrap | sh
  • Adds sourcing to .bashrc/.bash_profile to automatically inherit all .bash.d/*.sh environment enhancements for all technologies (see Inventory below)
  • Symlinks .* config dotfiles to $HOME for git, vim, top, htop, screen, tmux, editorconfig, Ansible, PostgreSQL .psqlrc etc. (only when they don’t already exist so there is no conflict with your own configs)
  • Installs OS package dependencies for all scripts (detects the OS and installs the right RPMs, Debs, Apk or Mac HomeBrew packages)
  • Installs Python packages
  • Installs AWS CLI

To only install package dependencies to run scripts, simply cd to the git clone directory and run make:

git clone https://github.com/HariSekhon/DevOps-Bash-tools bash-tools
cd bash-tools
make

make install sets your shell profile to source this repo. See Individual Setup Parts below for more install/uninstall options.

Inventory

  • Linux & Mac – curl OAuth / JWT, LDAP, find duplicate files, SSL certificate get/validate, URL encoding/decoding, Vagrant, advanced configurations:

    • .bashrc, .bash.d/*.sh, .gitconfig, .vimrc, .screenrc, .tmux.conf, .toprc, .gitignore…
  • AWS – Amazon Web Services – AWS account summary, lots of IAM reports, CIS Benchmark config hardening, EC2, ECR, EKS, Spot termination, S3 access logging, KMS key rotation info, SSM, CloudTrail, CloudWatch billing alarm with SNS notification topic and subscription for email alerts
  • GCP – Google Cloud Platform – massive GCP auto-inventory, scripts for GCE, GKE, GCR, Secret Manager, BigQuery, Cloud SQL, Cloud Scheduler, Terraform service account creation
  • Kubernetes – massive Kubernetes auto-inventory, cluster management scripts & tricks
  • Docker – Docker API, Dockerhub API, Quay.io API scripts
  • Databases – fast CLI wrappers, instant Docker sandboxes (PostgreSQL, MySQL, MariaDB, SQLite), SQL scripts, SQL script testers against all versions of a DB, advanced .psqlrc
  • Big Data & NoSQL – Kafka, Hadoop, HDFS, Hive, Impala, ZooKeeper, Cloudera Manager API & Cloudera Navigator API scripts
  • Git – GitHub, GitLab, Bitbucket, Azure DevOps – scripts for Git local & mirror management, GitHub, GitLab & BitBucket APIs
  • CI/CD – Continuous Integration / Continuous Delivery – API scripts & build pipeline configs for most major CI systems:

    • Jenkins, Concourse, GoCD, TeamCity – one-touch boot & build
    • Azure DevOps Pipelines, GitHub Actions Workflows, GitLab CI, BitBucket Pipelines, AppVeyor, BuildKite, Travis CI, Circle CI, Codefresh, CodeShip, Drone.io, Semaphore CI, Shippable, Wercker …
    • Terraform Cloud, Octopus Deploy
    • Checkov / Bridgecrew Cloud
  • Internet Services – Cloudflare, DataDog, Digital Ocean, GitGuardian, Jira, NGrok, Pingdom
  • Perl – Perl utilities & library management
  • Python – Python utilities & library management
  • Golang – Golang utilities
  • MP3 – MP3 metadata editing, grouping and ordering of albums and audiobooks
  • Spotify – 40+ Spotify API scripts for backups, managing playlists, track deduplication, URI conversion, search, add/delete, liked tracks, followed artists, top artists, top tracks etc.
  • More Linux & Mac – more systems administration scripts, package installation automation
  • Builds, Languages & Linting – programming language, build system & CI linting
  • Data Format Conversion & Validation – data validation scripts for CSV, JSON, Avro, Parquet, INI / Properties files (Java), LDAP LDIF, XML, YAML
  • Templates – Templates for AWS, GCP, Terraform, Docker, Jenkins, Cloud Build, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml etc.
  • Kubernetes Configs – Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

Linux & Mac

  • .* – dot conf files for lots of common software eg. advanced .vimrc, .gitconfig, massive .gitignore, .editorconfig, .screenrc, .tmux.conf etc.

    • .vimrc – contains many awesome vim tweaks, plus hotkeys for linting lots of different file types in place, including Python, Perl, Bash / Shell, Dockerfiles, JSON, YAML, XML, CSV, INI / Properties files, LDAP LDIF etc without leaving the editor!
    • .screenrc – fancy screen configuration including advanced colour bar, large history, hotkey reloading, auto-blanking etc.
    • .tmux.conf – fancy tmux configuration include advanced colour bar and plugins, settings, hotkey reloading etc.
    • Git:

      • .gitconfig – advanced Git configuration
      • .gitignore – extensive Git ignore of trivial files you shouldn’t commit
      • enhanced Git diffs
      • protections against committing AWS secret keys or merge conflict unresolved files
  • .bashrc – shell tuning and sourcing of .bash.d/*.sh
  • .bash.d/*.sh – thousands of lines of advanced bashrc code, aliases, functions and environment variables for:

    • Linux & Mac
    • SCM – Git, Mercurial, Svn
    • AWS
    • GCP
    • Docker
    • Kubernetes
    • Kafka
    • Vagrant
    • automatic GPG and SSH agent handling for handling encrypted private keys without re-entering passwords, and lazy evaluation to only prompt key load the first time SSH is called
    • and lots more – see .bash.d/README for a more detailed list
    • run make bash to link .bashrc/.bash_profile and the .* dot config files to your $HOME directory to auto-inherit everything
  • lib/*.sh – Bash utility libraries full of functions for
    Docker,
    environment,
    CI detection (Travis CI, Jenkins etc),
    port and HTTP url availability content checks etc.
    Sourced from all my other GitHub repos to make setting up Dockerized tests easier.
  • setup/install_*.sh – various simple to use installation scripts for common…


↑