A Fresh Look at SaltStack

By June 6, 2018Blog

I recently accepted a product management role for SaltStack, the IT orchestration and automation software company, after spending a good part of my career at a cyber security company assessing software vulnerabilities, evaluating configuration issues, detecting malware, and dealing with a host of other security-related issues.

Despite my background in IT security, I was new to SaltStack software, so I’ve been busy learning what makes SaltStack technology unique.

It’s been more than a month since I took the job, and here’s what I now know about the Salt platform. Its capabilities are the reason I became quickly comfortable making the transition to this new role, and I believe they are the same reasons you should be paying attention to this powerful but often underestimated company.

What is Salt?

One of the first things I learned was how Salt—and subsequently SaltStack—got its name. In 2011, Thomas Hatch, SaltStack technical founder, was tasked with managing large, complex data center infrastructures and none of the existing systems management or configuration management tools gave him the ability to quickly automate arbitrary tasks common to maintaining digital infrastructure. So he built a high-speed remote execution tool and decided to open source the project.

But the project needed a name. While watching Lord of the Rings, Thomas noticed that in a scene between Gimli the dwarf and the two hobbits Peregrin Took and Meriadoc Brandybuck, Gimli fixated on the salted pork the hobbits were eating. Tom thought to himself, “Well, everything is better with Salt,” and the name was born.

Just as almost everything is better with salt, SaltStack effectively adds spice to today’s modern IT infrastructure. Organizations are using Salt to discover, monitor, respond, orchestrate, automate, and secure assets across on-premises, hybrid, cloud, and IoT systems. There’s a vital difference that sets Salt apart from its competitors: Salt is designed to scale, and can easily handle tens of thousands of managed systems per master.  

The Salt project also has a massive and dynamic community, with 2500+ developers commiting to its code base and close to 100,000 commits to the project since its inception.

Architecture

The core of the Salt ecosystem consists of two critical components: the Salt Master and the Salt Minion. The other components play a supporting role to make the ecosystem work.

The following diagram provides a high-level architecture view of an example Salt architecture:

What Can Salt Do?

Over the last few years, IT infrastructure has become increasingly fragmented across on-premises, hybrid, cloud, and IoT assets. So, it’s become extremely complex for organizations to gain visibility into their infrastructure and understand where the assets are deployed and/or reside and—more importantly—what’s happening inside them.  

One of the first things Salt helps organizations with is asset inventory. Salt ships with four key modules to help organizations manage their assets.

Asset Inventory

Salt Minion

The Salt Minion is a Python-based endpoint that can be deployed on all major *nix and Windows operating systems. The Minions are connected to the Master, and can be asked to do a variety of tasks by the Master through an easy-to-understand YAML file or by directly running commands from the Master.

A Salt Master can run commands on the Minions as follows:

mehul@saltmaster:~$ sudo salt \* pkg.list_pkgs

A Salt Master could also request a Minion to configure the state of an asset in a certain way. For example, install ssh & curl packages in the example below.

install_packages:
 pkg.installed:
   - pkgs:
     - ssh
     - curl

In addition to providing accurate inventory of assets with Minions installed, Salt can also query public and private cloud providers for a list of assets deployed in the cloud infrastructure using Salt Cloud.

Salt Cloud

Salt Cloud integrates out of the box with more than two dozen public and private cloud providers, including all the big ones such as Google Cloud Platform, AWS, and Microsoft Azure. As a result, it’s easy to take inventory of all assets across providers with the help of a single command.

Here’s a quick output for a query against AWS for the list of all instances:

mehul@saltmaster:~$ sudo salt-cloud -Q

web01:
    ----------
     id: i-3ed6888c
     image: ami-1a013c5f
     name: web01
     private_ips: 172.31.7.224
     public_ips: 54.183.86.76
     size: t1.micro
     state: running

The same approach works against other cloud providers as well. The integrations are not limited to just querying cloud instances. They can also deploy or delete instances or even deploy a very complex application in the cloud.

Salt SSH (agentless)

Salt also supports agentless mode over SSH for assets that don’t have Salt Minions installed. The good part about the design of Salt SSH is it supports the same system normalization functions already supported by Salt Minions. For example, one can run the same set of commands supported by Salt Minions. Here is an example:

mehul@saltmaster:~$ sudo salt-ssh \* pkg.list_pkgs

Salt Proxy

Finally, on assets where agents can’t be deployed or it’s not possible to SSH in, Salt Proxy Minion provides an option to account for such devices. It’s designed to connect to network devices or IoT devices such as Philips Hue light bulbs.

Monitor (Beacons)

Salt Minions can be configured to monitor files, processes, services, and a host of other things. They can also generate events when certain criteria are met such as failed logins, unauthorized changes to critical files or processes, or even unexpected changes in CPU load or disk usage.

Here’s a simple example:

beacons:
 ps:
   - processes:
       mysql: stopped

beacons:
 diskusage:
   - /: 63%
   - /mnt/nfs: 90%

The Salt Beacon capability is the event generating element of SaltStack event-driven automation and is immensely valuable when coupled with the Salt Reactor.

Respond (Reactor)

Perhaps the most powerful feature of Salt is its ability to react to events and take actions using the Salt Reactor system. Salt Reactor can be configured to take predefined actions based on certain event criteria. For example, restart MySQL service if it stopped. This power helps organizations achieve a state of event-driven automation.

Here’s an overview of how it works:

  • The Salt Master and Salt Minions are connected to each other through an event bus.
  • When a Salt Master requests Salt Minions to perform an operation, such as run a command or install a package, the Salt Minion on completion registers the success or failure of that operation on the event bus.
  • Salt Beacons can also register their events on the event bus.
  • The Salt Reactor system can be configured to take one or more actions depending on the type of event such as, start a process if it is stopped or send an alert to Slack.

Here’s a relevant example in support of a DevOps workflow that sends a message to Slack once a build finishes:

alert slack:
 local.slack_notify.post_message
   - tgt: buildserver
   - kwarg:
       channel: "Development"
       api_key: peWcBiMOS9HrZG15peWcBiMOS9HrZG15"
       message: "Build {{ data['build_id'] }} finished with status: {{ data['status'] }}"

It’s extremely powerful, and can be called upon to react to a complex set of events. The only limiting factor is the creativity of the end user.

Orchestrate

The true power of Salt starts to become evident once the building blocks that make up Salt are used to orchestrate and deploy complex applications by running a single command. With orchestration capability, it is now possible to define the infrastructure as code.

Here’s an example of deploying web infrastructure with Redis installed:

deploy_redis_servers:
 salt.state:
   - tgt: '*redis*'
   - sls:
- redis

deploy_web_infra_servers:
 salt.state:
   - tgt: '*cweb*'
   - sls:
     - httpd
     - python
     - python.python-lxml
   - require:
     - Salt: deploy_redis_servers

deploy_project_group:
 salt.function:
   - name: group.add
   - tgt: '*'
   - arg:
     - web-team
   - require:
     - salt: deploy_web_infra_servers

This can be deployed with the run of a simple command:

mehul@saltmaster:~$ sudo salt-run state.orch orchestrate.web-infrastructure

Automate

Salt helps organizations automate routine IT jobs such as adding or removing users and updating servers with a single command. All of this is possible through its robust remote execution framework. The remote execution framework is made up of 477 execution modules that ship with Salt.

In the example below, “pkg” is the name of the execution module and “install” is the function being called within that module. Once the command is run it installs “vim” on all the available minions (the * represents all minions). Salt is intelligent enough to automatically detect the operating system, and will call “yum” for Red Hat OS and “apt-get” for Ubuntu under the hood to perform the function.   

Secure

Finally, Salt helps organizations secure their infrastructure. In my previous role, I spent a significant part of my career building software that assessed vulnerabilities and configuration drift, detected malware, and monitored a wide variety of other security issues. However, one thing the software products I managed couldn’t do was take remediation actions once the vulnerabilities or configuration drifts were discovered, or block threats by killing a process or closing down a port. Salt is perfectly designed to solve that part of the problem.

Some organizations I have spoken to are using Salt to satisfy 100% of their vulnerability patching needs as well as improve system hardening.

In some cases, it’s as trivial as running yum update:

mehul@saltmaster:~$ sudo salt ‘*’ cmd.run ‘yum update’

Or in the case of Windows it could be:

mehul@saltmaster:~$ sudo salt 'win*' win_wua.install_updates guid=['12345678-abcd-1234-abcd-1234567890ab', 'KB4022168']

This also happens to be the area of the SaltStack product line I will help manage. So, if you are using Salt in a professional setting, especially to satisfy a security use case like vulnerability remediation, autohealing configuration drift, detecting and responding to threats, I’d love to have a chat with you. Please drop us a note at product@saltstack.com.

Integrate

Salt is designed from the ground up to integrate with a variety of applications and services. The input to Salt is typically a YAML file, and output can be in YAML, json, or XML, output formats conducive for automation and integration. It can also be configured to return the data from jobs to a third-party data source (referred to as the Salt Returner) such as S3 or the PostgreSQL database for example. The combination of all these capabilities makes Salt a great candidate to play the role of conductor in this complex symphony of systems, code, and components we all now rely on as modern IT infrastructure.

I started working at SaltStack a few weeks ago, and in that short time it’s already become clear to me that Salt is an incredibly powerful piece of software that’s perfectly positioned to help organizations with its unique offering of capabilities. However, to really gain a full understanding of how it can help your business’s infrastructure, I recommend attending SaltConf18 which will be held September 10–13, 2018. There, you’ll have access to the industry’s most Salty and brilliant minds and you can get hands-on SaltStack training while learning best practices to spice up your automation.

Leave a Reply