What is the Elastic Stack?

May 21st, 2019

The Elastic Stack is a collection of freely downloadable products which enables engineers to implement search solutions. You may have heard of this called the "ELK" stack before. The Elastic Stack is an evolution of the ELK stack, but the underlying concepts haven't changed.

The stack components

Elasticsearch

Elasticsearch is a distributed search and analytics engine, built for speed, resilience, and scalability.

It's powered by a REST API that will be familiar to developers, and it offers a suite of SDKs so you can interact with Elasticsearch in your programming language of choice.

Logstash/Beats

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch or another repository.

Beats is the platform for single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.

Kibana

Kibana is your window into the Elastic Stack. It features powerful visualizations which allow you to create dashboards, discover trends, and make sense of your data.

It also has a suite of management features to help you administer the Elastic Stack.

What can I do with the Elastic Stack?

You can do a lot! It turns out that many common engineering problems are just search problems in disguise:

Logging - Elasticsearch is a fantastic repository for your application and infrastructure logs.

Security Analytics - Discover bad actors and anomalies on your network or application services.

Application Performance Metrics("APM") - with agents for many of the popular programming languages, you can use the Elastic Stack to monitor your application performance in real-time.

Custom search experiences - implement your own custom search in your website or app.

So much more! - The Elastic Stack is as powerful as it is flexible. With a thriving developer community, there is no shortage of new and novel solutions.

Getting started example

The best way to get started is to download Elasticsearch, and take it for a spin! We'll walk through a quick example of ingesting ngix logs into Elasticsearch in order to visualize them in Kibana.


1) Download

For this example, we will need Elasticsearch, Kibana, and Filebeat. Download and extract to a location of your choosing:

Download and extract Filebeat

Download and extract Elasticsearch

Download and extract Kibana


2) Start Elasticsearch

Start Elasticsearch from your terminal by running ./bin/elasticsearch from the directory you chose in step 1. This will start Elasticsearch with the default configuration, on port 9200. You can see it's running by going to http://localhost:9200


3) Start Kibana

Start Kibana from your terminal by running ./bin/kibana from the directory you chose in step 1. This will start Kibana with the default configuration, on port 5601. You can see it's running by going to http://localhost:5601.


4) Configure Filebeat

Now that Elasticsearch and Kibana are both running, it's time to configure Filebeat. Like the other components, we are sticking with the default configuration here.

The first step is to enable the nginx module. Filebeat ships with a number of modules which make configuration a breeze. To enable, run ./filebeat modules enable nginx.

Next, it's time to run the setup command: ./filebeat setup. The setup command will configure Elasticsearch to expect data from Filebeat, and it will install sample Dashboards into Kibana, so you can get started easier!

Note: If your nginx logs are not in a standard location on your machine, then you can configure Filebeat to look for them elsewhere.

Now, Filebeat is fully configured! It's time to start it by running ./filebeat -e from the terminal.


5) Explore your data!

Go you your Kibana dashboards at http://localhost:5601/app/kibana#/dashboards to get started! You should already have a sample dashboard available from Step 4.

- Written by Larry Gregory

Latest news
  • Coding stories to get you through hump day

    Come one come all and gather around the campfire. You’ve made it through that dreaded Monday, cruised through Tuesday on steam and now you’ve hit the Wednesday hump. Today, we’d like to help you get through it all with some coding horror stories. Share these with your colleagues to make their day and help them realize that a missing punctuation isn’t all that bad

    Jun 13th, 2019

  • CEO Survey Shows More Jobs For Upstate College Graduates

    Higher pay and rapid advancement retain talent in Upstate NY communities Upstate, New York, May 15, 2019 — The 3rd annual Upstate CEO Report, commissioned by Upstate Venture Connect, shows that fast-growing companies across our region have average salaries that are up to 60% higher than the average salary in a typical upstate metro area. “Scalable companies targeting national and global customers are distributed across our region,” stated UVC CEO Nasir Asli. “They don’t often show up on traditional economic development metrics, but clearly have an outsized impact on our communities.” Scalable companies are often in the newer industries and building products for national/global customers. Survey respondents in this category plan to hire 6-7 entry level people on average in 2019 and two thirds of those positions require a college degree.

    Jun 3rd, 2019 via https://uvc.org/ceo-survey-shows-more-jobs-for-upstate-college-graduates/

  • CEG TechAwards 2019

    Join hundreds of business professionals, entrepreneurs and innovators on June 27 for networking, awards ceremony and celebration with heavy hors d'oeuvres, cocktails, music & more!

    May 17th, 2019

  • TECHNOLOGY EMPLOYMENT EXPANDS ITS PRESENCE IN THE NATION’S WORKFORCE

    260,000-plus new jobs created in 2018 as 43 states grow their tech worker ranks, according to CompTIA Cyberstates 2019™ report Downers Grove, Ill. – Tech-related employment across the United States increased by more than 260,000 jobs in 2018 and the tech sector raised its contribution to the nation's economy, according to Cyberstates 2019™, the definitive guide to national, state and metropolitan area tech sector and tech workforce analytics published annually by CompTIA, the leading technology industry association.

    Apr 3rd, 2019 via https://www.comptia.org/about-us/newsroom/press-releases/2019/03/26/technology-employment-expands-its-presence-in-the-nation-s-workforce

  • Albany, NY | 2019 Top 100 Best Places to Live | Livability

    As the capital of New York, Albany has long been a hub of history, culture, art and more. Albany received an above-average rating in health, as the city is home to a number of hospitals (chief among them is St. Peters, which is considered high-performing in a number of advanced surgical procedures). Health education is also a focus for the city, with Albany Medical School producing two Nobel Prize winners and numerous leaders in medicine and research.

    Mar 18th, 2019 via https://livability.com/best-places/top-100-best-places-to-live/2019/ny/albany

Latest blog posts
  • 7 Interview Questions Every Aspiring JavaScript Developer Should Know

    Javascript is one of the most well-known programming languages in the world. It's used for anything from web applications to a server-side programming language. Looking for a job as a Javascript developer? We've compiled 7 questions you should already know the answers to. These questions are meant to get you prepared for your interview. Do more research to get even more prepared for the big day.

    Jun 3rd, 2019

  • What is the Elastic Stack?

    The Elastic Stack is a collection of freely downloadable products which enables engineers to implement search solutions. You may have heard of this called the "ELK" stack before. The Elastic Stack is an evolution of the ELK stack, but the underlying concepts haven't changed.

    May 21st, 2019

  • Spring Cleaning: Dust off that resume

    What if you’re asked to present or you want to pick up side work? If you’re happy at your job, you probably haven’t given any thought to your resume or LinkedIn profile. This is as important now as when you do decide to look for that next big opportunity. Just because you’re not looking right now, doesn’t mean that no one is paying attention to your current achievements.

    May 17th, 2019

  • Techstars Startup Weekend Returns

    "There's an app for that." You know that idea you had a while ago but didn't have the skills or resources to get it up and running? Ever wanted to join a driven team, ready to grind and with plans to create something great? The 7th Annual Techstars Startup Weekend will be held in Troy this time around at the IgniteU office on Mar 29, 2019. Presented by IgniteU NY, Sage School of Management, Troy Innovation Garage, In Focus Brands, CampusPro, Innovate 518, this years' event pulls in leaders from across the region who are both eager and willing to spend time with each team, building and improving upon their ideas.

    Mar 26th, 2019 via http://communities.techstars.com/events/14298

  • The state of gaming in the capital region

    The Capital Regions’ software community continues to grow in everything from AI to local user groups. Gaming is yet another software industry on the rise in the region. Startups and large companies are choosing the capital region to build and grow their applications. We’ve interviewed three leaders from local gaming companies who are growing the capital regions’ gaming community; Taro Omiya from Omiya Games, Clarke Foley from PUBG MadGlory, and Jamey Stevenson from Tech Valley Game Space.

    Mar 18th, 2019