What is the Elastic Stack?

May 21st, 2019

The Elastic Stack is a collection of freely downloadable products which enables engineers to implement search solutions. You may have heard of this called the "ELK" stack before. The Elastic Stack is an evolution of the ELK stack, but the underlying concepts haven't changed.

The stack components

Elasticsearch

Elasticsearch is a distributed search and analytics engine, built for speed, resilience, and scalability.

It's powered by a REST API that will be familiar to developers, and it offers a suite of SDKs so you can interact with Elasticsearch in your programming language of choice.

Logstash/Beats

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch or another repository.

Beats is the platform for single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.

Kibana

Kibana is your window into the Elastic Stack. It features powerful visualizations which allow you to create dashboards, discover trends, and make sense of your data.

It also has a suite of management features to help you administer the Elastic Stack.

What can I do with the Elastic Stack?

You can do a lot! It turns out that many common engineering problems are just search problems in disguise:

Logging - Elasticsearch is a fantastic repository for your application and infrastructure logs.

Security Analytics - Discover bad actors and anomalies on your network or application services.

Application Performance Metrics("APM") - with agents for many of the popular programming languages, you can use the Elastic Stack to monitor your application performance in real-time.

Custom search experiences - implement your own custom search in your website or app.

So much more! - The Elastic Stack is as powerful as it is flexible. With a thriving developer community, there is no shortage of new and novel solutions.

Getting started example

The best way to get started is to download Elasticsearch, and take it for a spin! We'll walk through a quick example of ingesting ngix logs into Elasticsearch in order to visualize them in Kibana.


1) Download

For this example, we will need Elasticsearch, Kibana, and Filebeat. Download and extract to a location of your choosing:

Download and extract Filebeat

Download and extract Elasticsearch

Download and extract Kibana


2) Start Elasticsearch

Start Elasticsearch from your terminal by running ./bin/elasticsearch from the directory you chose in step 1. This will start Elasticsearch with the default configuration, on port 9200. You can see it's running by going to http://localhost:9200


3) Start Kibana

Start Kibana from your terminal by running ./bin/kibana from the directory you chose in step 1. This will start Kibana with the default configuration, on port 5601. You can see it's running by going to http://localhost:5601.


4) Configure Filebeat

Now that Elasticsearch and Kibana are both running, it's time to configure Filebeat. Like the other components, we are sticking with the default configuration here.

The first step is to enable the nginx module. Filebeat ships with a number of modules which make configuration a breeze. To enable, run ./filebeat modules enable nginx.

Next, it's time to run the setup command: ./filebeat setup. The setup command will configure Elasticsearch to expect data from Filebeat, and it will install sample Dashboards into Kibana, so you can get started easier!

Note: If your nginx logs are not in a standard location on your machine, then you can configure Filebeat to look for them elsewhere.

Now, Filebeat is fully configured! It's time to start it by running ./filebeat -e from the terminal.


5) Explore your data!

Go you your Kibana dashboards at http://localhost:5601/app/kibana#/dashboards to get started! You should already have a sample dashboard available from Step 4.

- Written by Larry Gregory

Latest news
  • Data Says Demand for Tech Talent Continues

    An analysis of available data sets by CompTIA found that the U.S. technology sector employment continues to increase. In the first six months of 2019, more than 56,000 jobs were created for technology professionals, with another 13,500 jobs created in the month of June alone. This information comes from an examination of data from the U.S. Bureau of Labor Statistics’ “Employment Situation” report. Compared to last years’ report, so far this year an additional 6,700 jobs have been created as opposed to the first six months of 2018. “The mid-year tech employment report card shows many positives, reflecting the broad-based employer demand for tech talent,” says Tim Herbert, Executive Vice President for Research and Market Intelligence at CompTIA. “Businesses continue to make progress in adopting a range of emerging technologies, with hiring following suit.” Read More: https://mytechdecisions.com/compliance/data-says-demand-for-tech-talent-continues/

    Jul 12th, 2019 via https://mytechdecisions.com/compliance/data-says-demand-for-tech-talent-continues/

  • Alaant Hiring Index - Spring 2019

    After Strong Start to 2019, Hiring to Slow Amid Recruiting Challenges, According to Capital Region Employers Optimism remains high for future job growth, even as companies temper hiring expectations through the end of this year. Albany, NY – Hiring was robust in the first half of the year, and optimism over the long term remains high, but Capital Region employers are scaling back expectations for the second half of 2019 amid continued challenges in recruiting employees, according to the Spring 2019 Alaant Hiring Index released today. The index is compiled by Alaant Workforce Solutions, the Capital Region’s leading professional workforce services firm.

    Jun 19th, 2019 via https://www.alaant.com/blog-news/press-release-alaant-hiring-index-spring-2019

  • Coding stories to get you through hump day

    Come one come all and gather around the campfire. You’ve made it through that dreaded Monday, cruised through Tuesday on steam and now you’ve hit the Wednesday hump. Today, we’d like to help you get through it all with some coding horror stories. Share these with your colleagues to make their day and help them realize that a missing punctuation isn’t all that bad

    Jun 13th, 2019

  • CEO Survey Shows More Jobs For Upstate College Graduates

    Higher pay and rapid advancement retain talent in Upstate NY communities Upstate, New York, May 15, 2019 — The 3rd annual Upstate CEO Report, commissioned by Upstate Venture Connect, shows that fast-growing companies across our region have average salaries that are up to 60% higher than the average salary in a typical upstate metro area. “Scalable companies targeting national and global customers are distributed across our region,” stated UVC CEO Nasir Asli. “They don’t often show up on traditional economic development metrics, but clearly have an outsized impact on our communities.” Scalable companies are often in the newer industries and building products for national/global customers. Survey respondents in this category plan to hire 6-7 entry level people on average in 2019 and two thirds of those positions require a college degree.

    Jun 3rd, 2019 via https://uvc.org/ceo-survey-shows-more-jobs-for-upstate-college-graduates/

  • CEG TechAwards 2019

    Join hundreds of business professionals, entrepreneurs and innovators on June 27 for networking, awards ceremony and celebration with heavy hors d'oeuvres, cocktails, music & more!

    May 17th, 2019

Latest blog posts
  • SQL Saturday Coming to Albany!

    What is SQL Saturday? "SQLSaturday is a free training event for Microsoft Data Platform professionals and those wanting to learn about SQL Server, Business Intelligence and Analytics. This event will be held on Jul 20, 2019, at SUNY Albany, 1400 Washington Ave, Albany..."

    Jul 18th, 2019 via https://www.meetup.com/Capital-Area-SQL-Server-User-Group/events/257931962/

  • CEG Tech Awards 2019 Recap

    Last Thursday, at Rivers Casino Resort, several companies and leaders in the region attended the Center for Economic Growth (CEG)'s 23rd annual technology awards. The awards honor innovative and inspiring companies and individuals who help build the capital region.

    Jul 1st, 2019

  • So is QA important? YES!

    Imagine if updating your name on an app, sends your password to all your favorite contacts. Not only can you say goodbye to your privacy, you now have to update your passwords and hope that your friends aren’t too terrible. How do we avoid minor changes from having huge impacts? By QAing for “every possibility” you can think of and then some...and then some...

    Jun 24th, 2019

  • The state of SQL

    Microsoft SQL Serveris defined as “a relational database management system developed by Microsoft. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network.” There has been huge changes with Microsoft SQL and we’d like to get you up to speed.

    Jun 24th, 2019

  • 7 Interview Questions Every Aspiring JavaScript Developer Should Know

    Javascript is one of the most well-known programming languages in the world. It's used for anything from web applications to a server-side programming language. Looking for a job as a Javascript developer? We've compiled 7 questions you should already know the answers to. These questions are meant to get you prepared for your interview. Do more research to get even more prepared for the big day.

    Jun 3rd, 2019