What is the Elastic Stack?

May 21st, 2019

The Elastic Stack is a collection of freely downloadable products which enables engineers to implement search solutions. You may have heard of this called the "ELK" stack before. The Elastic Stack is an evolution of the ELK stack, but the underlying concepts haven't changed.

The stack components

Elasticsearch

Elasticsearch is a distributed search and analytics engine, built for speed, resilience, and scalability.

It's powered by a REST API that will be familiar to developers, and it offers a suite of SDKs so you can interact with Elasticsearch in your programming language of choice.

Logstash/Beats

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch or another repository.

Beats is the platform for single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.

Kibana

Kibana is your window into the Elastic Stack. It features powerful visualizations which allow you to create dashboards, discover trends, and make sense of your data.

It also has a suite of management features to help you administer the Elastic Stack.

What can I do with the Elastic Stack?

You can do a lot! It turns out that many common engineering problems are just search problems in disguise:

Logging - Elasticsearch is a fantastic repository for your application and infrastructure logs.

Security Analytics - Discover bad actors and anomalies on your network or application services.

Application Performance Metrics("APM") - with agents for many of the popular programming languages, you can use the Elastic Stack to monitor your application performance in real-time.

Custom search experiences - implement your own custom search in your website or app.

So much more! - The Elastic Stack is as powerful as it is flexible. With a thriving developer community, there is no shortage of new and novel solutions.

Getting started example

The best way to get started is to download Elasticsearch, and take it for a spin! We'll walk through a quick example of ingesting ngix logs into Elasticsearch in order to visualize them in Kibana.


1) Download

For this example, we will need Elasticsearch, Kibana, and Filebeat. Download and extract to a location of your choosing:

Download and extract Filebeat

Download and extract Elasticsearch

Download and extract Kibana


2) Start Elasticsearch

Start Elasticsearch from your terminal by running ./bin/elasticsearch from the directory you chose in step 1. This will start Elasticsearch with the default configuration, on port 9200. You can see it's running by going to http://localhost:9200


3) Start Kibana

Start Kibana from your terminal by running ./bin/kibana from the directory you chose in step 1. This will start Kibana with the default configuration, on port 5601. You can see it's running by going to http://localhost:5601.


4) Configure Filebeat

Now that Elasticsearch and Kibana are both running, it's time to configure Filebeat. Like the other components, we are sticking with the default configuration here.

The first step is to enable the nginx module. Filebeat ships with a number of modules which make configuration a breeze. To enable, run ./filebeat modules enable nginx.

Next, it's time to run the setup command: ./filebeat setup. The setup command will configure Elasticsearch to expect data from Filebeat, and it will install sample Dashboards into Kibana, so you can get started easier!

Note: If your nginx logs are not in a standard location on your machine, then you can configure Filebeat to look for them elsewhere.

Now, Filebeat is fully configured! It's time to start it by running ./filebeat -e from the terminal.


5) Explore your data!

Go you your Kibana dashboards at http://localhost:5601/app/kibana#/dashboards to get started! You should already have a sample dashboard available from Step 4.

- Written by Larry Gregory

Latest news
  • Indeed: Even non-tech companies are hiring more tech talent

    Dive Brief: The demand for tech jobs in non-tech industries is rising, a report from Indeed found. Tech-focused companies are still hiring plenty of tech workers, the data showed, but an increasing share of jobs at high-tech companies are going to workers in marketing, sales and other areas. Indeed described tech occupations as jobs involving "development or direct application of software, computers, or other information technology tools." Non-tech industries like finance, retail and energy have seen rapid growth in these type of workers. Software developer, a combination of two sub-occupations, applications and systems software, is currently the top in-demand tech job. Software developers account for 27% of all tech jobs, followed by broad computer occupations (including web admin and software quality assurance engineer), computer support specialists, information systems managers, information research scientists and programmers.

    Jul 30th, 2019 via https://5help.squarespace.com/hc/en-us/articles/219544088-Creating-a-login-page-or-link

  • Data Says Demand for Tech Talent Continues

    An analysis of available data sets by CompTIA found that the U.S. technology sector employment continues to increase. In the first six months of 2019, more than 56,000 jobs were created for technology professionals, with another 13,500 jobs created in the month of June alone. This information comes from an examination of data from the U.S. Bureau of Labor Statistics’ “Employment Situation” report. Compared to last years’ report, so far this year an additional 6,700 jobs have been created as opposed to the first six months of 2018. “The mid-year tech employment report card shows many positives, reflecting the broad-based employer demand for tech talent,” says Tim Herbert, Executive Vice President for Research and Market Intelligence at CompTIA. “Businesses continue to make progress in adopting a range of emerging technologies, with hiring following suit.” Read More: https://mytechdecisions.com/compliance/data-says-demand-for-tech-talent-continues/

    Jul 12th, 2019 via https://mytechdecisions.com/compliance/data-says-demand-for-tech-talent-continues/

  • Alaant Hiring Index - Spring 2019

    After Strong Start to 2019, Hiring to Slow Amid Recruiting Challenges, According to Capital Region Employers Optimism remains high for future job growth, even as companies temper hiring expectations through the end of this year. Albany, NY – Hiring was robust in the first half of the year, and optimism over the long term remains high, but Capital Region employers are scaling back expectations for the second half of 2019 amid continued challenges in recruiting employees, according to the Spring 2019 Alaant Hiring Index released today. The index is compiled by Alaant Workforce Solutions, the Capital Region’s leading professional workforce services firm.

    Jun 19th, 2019 via https://www.alaant.com/blog-news/press-release-alaant-hiring-index-spring-2019

  • Coding stories to get you through hump day

    Come one come all and gather around the campfire. You’ve made it through that dreaded Monday, cruised through Tuesday on steam and now you’ve hit the Wednesday hump. Today, we’d like to help you get through it all with some coding horror stories. Share these with your colleagues to make their day and help them realize that a missing punctuation isn’t all that bad

    Jun 13th, 2019

  • CEO Survey Shows More Jobs For Upstate College Graduates

    Higher pay and rapid advancement retain talent in Upstate NY communities Upstate, New York, May 15, 2019 — The 3rd annual Upstate CEO Report, commissioned by Upstate Venture Connect, shows that fast-growing companies across our region have average salaries that are up to 60% higher than the average salary in a typical upstate metro area. “Scalable companies targeting national and global customers are distributed across our region,” stated UVC CEO Nasir Asli. “They don’t often show up on traditional economic development metrics, but clearly have an outsized impact on our communities.” Scalable companies are often in the newer industries and building products for national/global customers. Survey respondents in this category plan to hire 6-7 entry level people on average in 2019 and two thirds of those positions require a college degree.

    Jun 3rd, 2019 via https://uvc.org/ceo-survey-shows-more-jobs-for-upstate-college-graduates/

Latest blog posts
  • Flutter vs React Native

    As more and more companies and people embrace modern technology as a norm or even necessity to they daily success, the demand for mobile cross-platform mobile application continues to grow. In order to stay ahead of the curve of leading-edge technology, many look to development frameworks for quick turnaround times for their applications. Developers also look here to make their development process easy and efficient.

    Aug 19th, 2019 via https://www.thedroidsonroids.com/blog/flutter-vs-react-native-what-to-choose-in-2019

  • Is computer science the right major for you?

    A job filled with opportunities, a decent salary, a constant flow of new information, and a plethora of options. That is the life you’ll live after studying computer science and following the road down the computer scientist lifestyle. At least those are the good things. With that comes hours at a computer, eye strain, headaches, dealing with clients, and so much more. Yes, computer science is a popular major but is it the right major for you?

    Jul 29th, 2019

  • Meet the 2019 IgniteU teams

    Looking for an entrepreneurial hub right in the heart of Troy? Looking to ignite (pun intended) your startup and take it to the next level? IgniteU is exactly what you’re looking for. This summer, eight great startups led by brave entrepreneurs will be going through the IgniteU summer accelerator where they will be taking their companies to the next level. In this blog, we’ll introduce you to all eight startups AND tell you about an opportunity to see them at the final pitch of the year. We urge you to read through each startups’ description, follow them on social media, and join and network with several leaders, entrepreneurs, and companies in the region at the final event. P.S. There are eight companies so we added some puns to making reading a bit more bearable.

    Jul 22nd, 2019

  • SQL Saturday Coming to Albany!

    What is SQL Saturday? "SQLSaturday is a free training event for Microsoft Data Platform professionals and those wanting to learn about SQL Server, Business Intelligence and Analytics. This event will be held on Jul 20, 2019, at SUNY Albany, 1400 Washington Ave, Albany..."

    Jul 18th, 2019 via https://www.meetup.com/Capital-Area-SQL-Server-User-Group/events/257931962/

  • CEG Tech Awards 2019 Recap

    Last Thursday, at Rivers Casino Resort, several companies and leaders in the region attended the Center for Economic Growth (CEG)'s 23rd annual technology awards. The awards honor innovative and inspiring companies and individuals who help build the capital region.

    Jul 1st, 2019