Welcome!

Apache Authors: Elizabeth White, Pat Romanski, Liz McMillan, Christopher Harrold, Janakiram MSV

Related Topics: @DevOpsSummit, Apache

@DevOpsSummit: Blog Post

Active vs. Passive Server Monitoring By @HoardingInfo | @DevOpsSummit [#DevOps]

It's important to determine if you are only responding to what the logs tell you about the past

Active vs. Passive Server Monitoring

By Chris Riley

Server monitoring is a requirement, not a choice. It is used for your entire software stack, web-based enterprise suites, custom applications, e-commerce sites, local area networks, etc. Unmonitored servers are lost opportunities for optimization, difficult to maintain, more unpredictable, and more prone to failure.

While it is very likely that your team has a log analysis initiative, it's important to determine if you are only responding to what the logs tell you about the past, or are you planning ahead based on the valuable log data you are monitoring and analyzing?

active-vs-passive-server-monitoring2

There are two basic approaches to server monitoring: Passive and Active. They are as much a state of mind as a process. And there are significant differences in the kinds of value each provide; each has its own advantages and disadvantages.

Passive Monitoring

Passive server monitoring looks at real-world historical performance by monitoring actual log-ins, site hits, clicks, requests for data, and other server transactions. When it comes to addressing issues in the system, the team will review historical log data, and from them that analyze the logs to troubleshoot and pinpoint issues. This was previously done with a manual pull of logs. While this helps developers identify where issues are,  using a powerful t modern log analysis service to simply  automate an existing process is a waste.

Passive server monitoring only shows how your server handles existing conditions, but it may not give you much insight into how your server will deal with future ones. For example, if one of the components of the system, a database server, is likely to be overloaded when the load rate of change is reached. This is not going to be clear when server log data has already been recorded, unless your team is willing to stare at a graph in real-time, 24/7...which has been nearly the case in some NOC operations I have witnessed.

Active Monitoring

The most effective way to get past these limits is by using active server monitoring. Active monitoring is the approach that leverages smart recognition algorithms to take current log data and use it to predict future states. This is done by some complex statistics (way over my head) that compare real-time to previous conditions, or past issues. For example it leverages anomaly detection, steady state analysis, and trending capabilities to predict that a workload is about to hit its max capacity. Or there is a sudden decrease in external network-received packets, a sign of public web degradation.

Besides finding out what is possibly going to happen. It also helps to avoid the time spent on log deep dives. Issues will sometimes still pass you by, and you will still need to take a deeper look, but because information is pushed to you, some of the work is already done, and you can avoid the log hunt.

Oh and it can help the product and dev team from an architectural standpoint. If, for example, a key page is being accessed infrequently, or if a specific link to that page is rarely used, it may indicate a problem with the design of the referring page, or with one of the links leading to that page. A close look at the log can also tell you whether certain pages are being accessed more often than expected  -  which can be a sign that the information on those pages should be displayed or linked more prominently.

Any Form of Server Monitoring is Better Than None

Log analysis tools are the heart of both approaches. Log analysis can indicate unusual activity which might slip past and already overloaded team. Another serious case is security. A series of attempted page hits that produce "page not found" or access denied" errors, for example, could just be coming from a bad external link  -  or they could be signs of an attacker probing your site. HTTP request that are pegging a server process could be a sign that a denial of service attack has begun.

It is hard to make the shift. Why? Not because you and your team are not interested in thinking ahead. But more so because many operations are entrenched in existing processes that are also reactive. And sometimes teams are just unaware that their tool can provide this type of functionality. Until one day it does it automatically for you, and you have a pleasant surprise.

Active server monitoring can mean the difference between preventing problems before they get a chance to happen, or rushing to catch up with trouble after it happens. And they are the difference between a modern version of an old process, and moving forward to a modern software delivery pipeline.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

IoT & Smart Cities Stories
Early Bird Registration Discount Expires on August 31, 2018 Conference Registration Link ▸ HERE. Pick from all 200 sessions in all 10 tracks, plus 22 Keynotes & General Sessions! Lunch is served two days. EXPIRES AUGUST 31, 2018. Ticket prices: ($1,295-Aug 31) ($1,495-Oct 31) ($1,995-Nov 12) ($2,500-Walk-in)
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...