What is the Elastic Stack?

May 21st, 2019

The Elastic Stack is a collection of freely downloadable products which enables engineers to implement search solutions. You may have heard of this called the "ELK" stack before. The Elastic Stack is an evolution of the ELK stack, but the underlying concepts haven't changed.

The stack components

Elasticsearch

Elasticsearch is a distributed search and analytics engine, built for speed, resilience, and scalability.

It's powered by a REST API that will be familiar to developers, and it offers a suite of SDKs so you can interact with Elasticsearch in your programming language of choice.

Logstash/Beats

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch or another repository.

Beats is the platform for single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.

Kibana

Kibana is your window into the Elastic Stack. It features powerful visualizations which allow you to create dashboards, discover trends, and make sense of your data.

It also has a suite of management features to help you administer the Elastic Stack.

What can I do with the Elastic Stack?

You can do a lot! It turns out that many common engineering problems are just search problems in disguise:

Logging - Elasticsearch is a fantastic repository for your application and infrastructure logs.

Security Analytics - Discover bad actors and anomalies on your network or application services.

Application Performance Metrics("APM") - with agents for many of the popular programming languages, you can use the Elastic Stack to monitor your application performance in real-time.

Custom search experiences - implement your own custom search in your website or app.

So much more! - The Elastic Stack is as powerful as it is flexible. With a thriving developer community, there is no shortage of new and novel solutions.

Getting started example

The best way to get started is to download Elasticsearch, and take it for a spin! We'll walk through a quick example of ingesting ngix logs into Elasticsearch in order to visualize them in Kibana.


1) Download

For this example, we will need Elasticsearch, Kibana, and Filebeat. Download and extract to a location of your choosing:

Download and extract Filebeat

Download and extract Elasticsearch

Download and extract Kibana


2) Start Elasticsearch

Start Elasticsearch from your terminal by running ./bin/elasticsearch from the directory you chose in step 1. This will start Elasticsearch with the default configuration, on port 9200. You can see it's running by going to http://localhost:9200


3) Start Kibana

Start Kibana from your terminal by running ./bin/kibana from the directory you chose in step 1. This will start Kibana with the default configuration, on port 5601. You can see it's running by going to http://localhost:5601.


4) Configure Filebeat

Now that Elasticsearch and Kibana are both running, it's time to configure Filebeat. Like the other components, we are sticking with the default configuration here.

The first step is to enable the nginx module. Filebeat ships with a number of modules which make configuration a breeze. To enable, run ./filebeat modules enable nginx.

Next, it's time to run the setup command: ./filebeat setup. The setup command will configure Elasticsearch to expect data from Filebeat, and it will install sample Dashboards into Kibana, so you can get started easier!

Note: If your nginx logs are not in a standard location on your machine, then you can configure Filebeat to look for them elsewhere.

Now, Filebeat is fully configured! It's time to start it by running ./filebeat -e from the terminal.


5) Explore your data!

Go you your Kibana dashboards at http://localhost:5601/app/kibana#/dashboards to get started! You should already have a sample dashboard available from Step 4.

- Written by Larry Gregory

Latest news
  • 2019 Most In-Demand Programming Languages + Local Jobs in them

    Computer Science has been one of the most sought after majors in the most recent years and (other than the love of technology) students are driven to it by salary. With salaries of over $80,000, who wouldn’t want to be at a computer coding away. Now you not only get to stay on the computer for hours, you actually get paid for it!

    Sep 30th, 2019 via https://www.codeplatoon.org/the-best-paying-and-most-in-demand-programming-languages-in-2019/

  • How this Texas-based tech company is finding big growth in New York

    Tyler Technologies came to the Albany market in 2008 when it acquired Versatrans Solutions, which developed K-12 school bus routing software. Since then, its client base has more than doubled and revenues have tripled. Its Albany-area staff has grown from 60 employees to more than 130.

    Sep 18th, 2019

  • Indeed: Even non-tech companies are hiring more tech talent

    Dive Brief: The demand for tech jobs in non-tech industries is rising, a report from Indeed found. Tech-focused companies are still hiring plenty of tech workers, the data showed, but an increasing share of jobs at high-tech companies are going to workers in marketing, sales and other areas. Indeed described tech occupations as jobs involving "development or direct application of software, computers, or other information technology tools." Non-tech industries like finance, retail and energy have seen rapid growth in these type of workers. Software developer, a combination of two sub-occupations, applications and systems software, is currently the top in-demand tech job. Software developers account for 27% of all tech jobs, followed by broad computer occupations (including web admin and software quality assurance engineer), computer support specialists, information systems managers, information research scientists and programmers.

    Jul 30th, 2019 via https://5help.squarespace.com/hc/en-us/articles/219544088-Creating-a-login-page-or-link

  • Data Says Demand for Tech Talent Continues

    An analysis of available data sets by CompTIA found that the U.S. technology sector employment continues to increase. In the first six months of 2019, more than 56,000 jobs were created for technology professionals, with another 13,500 jobs created in the month of June alone. This information comes from an examination of data from the U.S. Bureau of Labor Statistics’ “Employment Situation” report. Compared to last years’ report, so far this year an additional 6,700 jobs have been created as opposed to the first six months of 2018. “The mid-year tech employment report card shows many positives, reflecting the broad-based employer demand for tech talent,” says Tim Herbert, Executive Vice President for Research and Market Intelligence at CompTIA. “Businesses continue to make progress in adopting a range of emerging technologies, with hiring following suit.” Read More: https://mytechdecisions.com/compliance/data-says-demand-for-tech-talent-continues/

    Jul 12th, 2019 via https://mytechdecisions.com/compliance/data-says-demand-for-tech-talent-continues/

  • Alaant Hiring Index - Spring 2019

    After Strong Start to 2019, Hiring to Slow Amid Recruiting Challenges, According to Capital Region Employers Optimism remains high for future job growth, even as companies temper hiring expectations through the end of this year. Albany, NY – Hiring was robust in the first half of the year, and optimism over the long term remains high, but Capital Region employers are scaling back expectations for the second half of 2019 amid continued challenges in recruiting employees, according to the Spring 2019 Alaant Hiring Index released today. The index is compiled by Alaant Workforce Solutions, the Capital Region’s leading professional workforce services firm.

    Jun 19th, 2019 via https://www.alaant.com/blog-news/press-release-alaant-hiring-index-spring-2019

Latest blog posts
  • Sitting at your desk (The right way)

    As members of the software community, we spend a lot of time typing away on our computers and staring at a computer screen. Over-time this can lead to some pretty nasty pains throughout the body. Below are three quick things to keep in mind when sitting at your desk for a long time.

    Oct 14th, 2019

  • 4 Code Cleaning Tips to Follow

    Indents, tabs, newlines, brackets... One of the main reasons for clean and easy to read code is to make collaboration and maintenance seamless. A clean set of code is easy to traverse and makes collaboration seamless. The following are just four tips to keep your code understandable and sensible.

    Oct 7th, 2019

  • SaaS vs. SaaP

    SaaS vs. SaaP What are they? So first let's break down the acronym. Software as a Product (SaaP). Software as a Service (SaaS). 

    Sep 23rd, 2019

  • Moving to a big city? Stay in the 518...

    “It now costs $350,000 a year to live a middle-class lifestyle in a big city -- here’s a sad breakdown of why”. This recent post from CNBC breaks down the truth behind those large salaries that attract so many people to these large cities. Although “less than 5% of households earn $350,000 or more a year” that amount of money will go quickly when you’re living in an expensive city like NYC for example.

    Sep 16th, 2019

  • The Forth DevFest Conference in the Region

    The Capital Region Google Developer Group DevFest conference is back is back for its fourth year! This conference targets serving the local and regional software engineer, developer, architect, product manager, and student community. Topics include: mobile, web apps, AR/VR, IoT, machine learning and cloud computing.

    Sep 9th, 2019