Taking Elastic Agent Out For A Test Drive

Intro:

The latest version of the Elastic Stack, 7.9, has bombarded us with great new features, free endpoint security, and some sleek changes to the Kibana UI. In this article, I will show you one of the features I am most excited about since it was a topic a client brought up to me over a year ago.

In the beginning, there was Logstash and it served its purposes. Then came Beat Modules that could replace Logstash for most use cases. The problem with Beats is that there were so many of them. When designing an architecture for a Security-as-a-Service product I was asked a very simple question. “Can we install these Beats with a single administration panel remotely, and with fewer steps for the admin?” At the time I immediately thought of a platform like Ansible but realized I would also need a complex control agent system to manage it, not to mention a frontend(I am very much backend, and so this worried me a bit). Luckily I happened to be attending one of the last Elastic-On-Tours before Covid-19 and asked this question in a small discussion group. The answer was to save my effort and just wait for something called “Fleet”.

Well in less than a year I finally know what Fleet is, and the question has been turned into a feature in Beta status. The key to this mighty management problem is the Elastic Agent. It is a single agent that is installed on the system you wish to ingest data from. The configuration is very similar to a Beat, so setting them up is pretty simple. The fantastic part is that this little agent now manages all other Beats and/or Endpoint agents that are going to be employed on that system, and what modules they will be using.

The Elastic Agent has some limitations as its abilities get rolled out once they have been tested. I should also warn you that this should not be used in production yet, but I just love shiny new toys. The best part of the Agent in my opinion is that it can handle Filebeat, Metricbeat, and Elastic Endpoint Security(We will cover that in another article). The agent can either be managed locally or from a dedicated and secured Kibana Instance.

Fleet is the new flagship of Ingest Management for Elastic. From the control panel in Kibana you can remotely configure Elastic Agents for whatever tasks you need them to do. Let’s walk through this process step by step.

First, you will need to go to the management tab in Kibana and select the ingest manager:

The Ingest Manager:

The overview tab of the ingest manager provides you information on the different aspects of the agents providing data collection. The first tab is integrations. An integration designates what type of data will be collected. For example, you will find your familiar modules that bring in various logs, metrics of various system types, and even Elastic Endpoint Security. The visual layout and descriptions make choosing the integrations very simple and intuitive. The agent configuration tab makes it possible to create various config files for different types of deployments. The brilliant part about this is it makes it easy to organize integrations based on purpose not on what Beats happen to be installed. Also, the configurations can be hot-loaded, meaning the agent does not need to be restarted on the host when a change is made. Fleet is the central screen where all agents can be managed and deployment data is available. Finally, datasets will contain various metrics that can be reported on the agents themselves. Some of these features are similar to older Central Beats Management panels, but with the big exception that you only need to install one agent on a system, and the integrations are much easier to choose and manage.

Learn by Doing: Setting Up an Agent Configuration

• Choose the configuration tab and click on the ‘create agent’ button.
• You can give the configuration a specific name and description.
• I chose to call my configuration Windows Desktop. I described it as the configuration I will use on Windows Desktops.

• Next, we can add integrations
• By default, the system integration collects metrics related to the general operation
• There are many choices with descriptions and pictures
• For a Windows Desktop, I have chosen the Windows Integration(Specific Metricbeat data related to Windows) and Elastic Endpoint Security.
• Unfortunately, the first beta roll-out does not include Winlogbeat support, but I am sure it will be there in the future. The docs mention not supporting processors, and Winlogbeat uses javascript processors amongst other things.
• Also, note that on the configuration management screen you can both edit and/or delete the configuration.

Learn by Doing: Setting Up Fleet

• Elastic Agents can be configured locally, but in my opinion, the real power is utilizing Fleet to manage them.
• The main requirement for using Fleet is setting up a trusted communication route between the agent and Kibana
• In this example, I am using a deployment on Elastic Cloud and so it is automatically configured this way.
• The good news is those same security features are now free for local deployment as well.
• https://www.elastic.co/guide/en/ingest-management/current/run-elastic-agent.html has the steps to secure your stack for the usage of Fleet.
• Once on the Fleet screen, you want to add an agent and assign the Windows Desktop Configuration to that agent.
• Once this is done you can get the enrollment token you will need to configure the agent on the Windows System.
• These are the same steps for any agent and configuration. I just chose to set up a Windows security configuration for this example.
• Note in the figure below I have already started having the agent send data, and so it now names the agent according to the Host it is on automatically.

Learn by Doing: Installing and Configuring the Elastic Agent

• Download the Windows Elastic Agent on the following page, https://www.elastic.co/downloads/beats/elastic-agent
• There are other downloads for various operating systems just like with beats
• Extract the contents into the “C:\Program Files” directory.
• Rename the folder “Elastic-Agent”
• Open an administrator PowerShell terminal and enter the following commands:

cd “C:\Program Files\Elastic-Agent”

.\elastic-agent.exe enroll KIBANA_URL ENROLLMENT_KEY

Set-ExecutionPolicy Unrestricted

.\install-service-elastic-agent.ps1

Start-Service elastic-agent

• Remember that new integrations or other configuration changes can be made and will automatically be sent to the agent by Fleet without restarting the agent.

Conclusion:

The Elastic Agent has been referred to as “One Agent to Rule Them All” in the great Lord of The Rings style and indeed it is. I have also seen it referred to as “Uber Beats” because of its awesome under-the-hood management of Beats. Quite frankly the combination of the new agent and the Kibana Fleet application has completely changed the game on how we architect our Stacks. It is amazing to think that some of us started in the days of searching Github repositories for the various Logstash configurations, to manually installing and configuring beats and now literally having an integration menu as if the Elastic Stack was a fast-food restaurant. Well, I am definitely happy to tell you that what I have demonstrated is only the surface of what will be released later, and so we can definitely expect the Elastic Agent to be “Super Sized” in later releases.