↓Skip to main content
IT Blog IT Blog
  1. All Posts/

splunk-n-box

Tools Shell

**** IMPORTANT DOCKER NOTE ****

As of Feb 29,2020 there is a confirmed IP aliasing bug in the latest docker release that applies to both Mac and Windows Docker Desktop versions. This bug breaks Splunk N’ A Box.
Downgrading to the latest docker 2.1.x version (2.1.0.5) solves the problem. Here are the direct download links.
Mac: https://download.docker.com/mac/stable/40693/Docker.dmg
Windows: https://download.docker.com/win/stable/40693/Docker%20Desktop%20Installer.exe

Videos:

🔗 Splunk N’ Box full presentation (.conf2017): https://youtu.be/tXeFwWTXtD4
🔗 Run Splunk n’ Box on USB stick: https://youtu.be/qTAS1gvIGxM

Introduction:

Have you ever wanted to create a multi-site cluster environment in your lab but you don’t have the resources for it? Have you ever wondered how does bucket replaction impact the file system level? Have you ever wanted to create a portable Splunk classroom but it is cost prohibitive? How about changing critical configuration without worrying about messing up your production environment? If you are like me; you must have dealt with similar challenges.
Like with most people, you probably attempted to solve the problem by either throwing more hardware at it or by using a VM technology that does not scale well without additional resources (and cost). Well, I have a solution for you! But before that, I would like to welcome you to the world of DOCKER! The game changer that brought micro services to reality. Imagine that with a click of a button you can create 3-site cluster; each site has 3-SHs and 10-IDXs. Or maybe just instantly create a portable lab environment for training purposes. You may have heard of Docker, or you may even experiment with it trying to figure out how can I use it to help my Splunk needs. But learning Docker technology by itself is not helpful unless used in a context of a specific app like Splunk. To help my customers (and myself) I have created a wrapper bash script (~4000 lines) to manage Splunk instances builds. The script will allow you to create a pre-configured large number of Splunk infrastructure components without having to learn a single docker command and with minimal resources requirements.
In my small test environment, I was able to quickly bring upward of 40+ Splunk Docker containers for a classroom lab using low powered Intel NUC device (i3 16GB ram, 128G SSD). What’s impressive about Docker is the resource utilization on the docker-host is tiny compared to a VM based build. I need to emphasize the fact that I have not tested builds under heavy load (either user traffic or data ingestion). However, I believe it is just a matter of sizing the hardware appropriately.

Feature list:

  • Menu driven and user friendly colorful interface to manage splunk docker containers.
  • Continues status bar feedback communicating docker host health and state.
  • MacOS run can can utitlize voice (siri). Check the CLI options.
  • Tunning speed (useful with fast CPU hosts).
  • Extensive error checking and validation.
  • Support for multiple Splunk versions (images).
  • Adaptive load control during cluster build (throttle execution if exceeds 4 x cores).
  • Built-in dynamic hostnames and IPs allocation (no need for proxy container like NGINX)
  • Automatically create & configure large number of Splunk hosts very fast
    (under 10 mins for fully running single site cluster).
  • Different levels of logging (show docker commands executed).
  • Fully configured multi & single site cluster builds (including LM,CM, DEP, DMC servers).
  • Manual and automatic cluster builds.
  • Modular design that can easily be converted to a higher-level language like Python.
  • Custom login screen (helpful for lab & Search Parties scenarios).
  • Low resources requirements compared to VM based solutions.
  • Eliminate the need to learn docker CLI (but you should).
  • MacOS & Linux support.
  • Works with windows10 WSL (Windows Subsystem for Linux) Ubuntu bash.
  • Automatic online script upgrade (with version check).
  • AWS EC2 aware (shows the NATed IPs)

How to install?

Source code is posted here: https://github.com/mhassan2/splunk-n-box

start by expanding your terminal display (iTerm) to maximum columns/rows

cd ~
git clone https://github.com/mhassan2/splunk-n-box
cd splunk-n-box
cp <your_valid_lic_files> splunk_licenses/
./splunknbox.sh

How does it work?

Once you have your Ubuntu up and running, please follow the instructions for installing Docker https://docs.docker.com/engine/installation/linux/ubuntulinux/
Please be aware that Ubuntu 14.04 did not work very well for me. There is a bug around mounting docker volumes. Your mileage may vary if you decide to use CentOS or equal Linux distribution.
For OSX see https://docs.docker.com/engine/installation/mac/
When you run the scripts for the first time, it will check to see if you have any IP aliases available (the range specified in the script). If not; then it will configure IP aliases 192.168.1.100-254. The aliased IPs will be automatically mapped, at container creation time, to the internal docker IP space (172.18.0.0/24). You should be able to point your browser to any NATed IP on port 8000, and that will get you directly to the container. During my research, I haven’t seen many people using this technique, and they mostly opt for changing the ports or using a proxy container. My approach is to keep the standard Splunk ports (8000, 8089, 9997, etc.) and use iptable NATs to make the containers visible to the outside world. This trick will save you a lot of headaches when dealing with creating a large number of Splunk containers (aka hosts). Running under OSX, I used private network segment 10.0.0.0/24. The assumption here is you don’t need to NAT to the outside world and everything will be local to your MAC laptop. Windows and OSX do not support Linux c-groups natively. Therefore there is an additional layer of virtualization required, which will impact performance.

Splunk image(s):

All splunk images (with multiple versions) are pre-built and posted to docker hub. The script will utilize this URL during execution to download the desired image

https://hub.docker.com/r/splunknboxk/splunk_x.x.x/

Linux installation note:

For different linux distributions/versions see: https://docs.docker.com/engine/installation/
if you get this message when running the script
WARNING: No swap limit support
WARNING: No swap limit support
WARNING: No swap limit support
try this fix posted here: moby/moby#4250

  1. /etc/default/grub: 
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1"
sudo update-grub && sudo reboot

If you want the docker-host to be able to resolve host IPs (optional) install dnsmasq (google for your Linux flavor).
Change DNSSERVER=”192.168.2.100″ to point the caching DNS server. This does not work on OSX yet!

Windows10 installtion note:

1- Install Windows 10 WSL (Windows Subsystem for Linux)
https://msdn.microsoft.com/en-us/commandline/wsl/install_guide
2- Install Docker for windows
https://docs.docker.com/docker-for-windows/install/
3- Add windows loopback KM-TEST (not enabled by default)
https://technet.microsoft.com/en-us/library/cc708322(v=ws.10).aspx
https://www.pingzic.com/how-to-enable-loopback-adapter-in-windows-10/
4- Add IP aliases using cmd.exe (running as admin). Future bash.exe fixed ifconfig problem under WSL, so this step can be accomplished from the bash session
http://www.ibm.com/support/knowledgecenter/SSNKWF_8.0.0/com.ibm.rational.test.lt.doc/topics/tconfigip_win.html

netsh -c Interface ip add address name="KM-TEST" addr=10.0.0.101 mask=255.255.0.0

Repeat the above step for IPs 10.0.0.101-200
To remove IP aliases
The ntcmds.chm file, typically located in C:WINDOWSHelp, contains more details about the netsh command. When you are finished with the IP aliases, use the following command to remove them:

netsh -c Interface ip delete address name="KM-TEST" addr=x.x.x.x   (repeat for all IPs)

5- Add export DOCKER_HOST=tcp://127.0.0.1:2375 to .bashrc then resource .bashrc
6- Restart…



↑