How BI tools like Splunk are crucial to any organization.

Tools like Splunk are a fantastic example of what real time results can do for you. It literally allows you to drill down to the packet level of what is happening when, where and by whom in your organization. In the age of big data, log management is becoming an absolute necessity, as developers, operations, and, yes, even senior management have to deal with and process huge amounts of machine-generated data. Many organizations have turned to Splunk, a pioneer in the space, to help manage the rising tide of log data – but Splunk can get really, really expensive, FAST.

While there still is not a single, all purpose alternative to Splunk that is as robust and stable, there are several tools that can be used to replicate much of its functionality. In fact Booking.com SysAdmin Brad Lhotsky documented his quest to build his own central log management system using only open source software.

Of course, his blog entry contains much deeper technical insight, but at the high level, he broke his solution down into three components: Log centralization (rsyslog), log management (logstash/Kibana) and log visualization (Graphite).

Rsyslog was tapped for log centralization over similarly popular alternative syslog-ng because the former offers guaranteed delivery and encrypted transfer in the open source edition – two features that Lhotsky says are becoming of increased importance to regulatory compliance auditors. With rsyslog, Lhotsky was able to build a reliable way to transport event logs from Unix hosts to a central repository.

This is where Lhotsky starts entering Splunk’s territory, calling the company “the 1,000 lb Gorilla in the room.” But in lieu of Splunk, Lhotsky writes that he took the MongoDB-powered Graylog2 for a test drive before settling on logstash. Graylog2 is great, he says, but suggests that its ElasticSearch indexing scheme is “broken,” and if you have to keep a large amount of logs around for compliance reasons, you’re going to take a performance hit. Lhotsky goes so far as to speculate that it’s because Graylog2 only implemented ElasticSearch for, well, search fairly late in the game.

On the other side of the coin, logstash also uses ElasticSearch, but with far more of a focus on scalability, inputs, filter and outputs. The cost, Lhotsky writes, is a polished front-end. Enter Kibana, a PHP front-end for logstash that takes the ElasticSearch indexes and adds a front-end for search and analysis, making the whole platform a lot more usable.

“Kibana fills the gap with the Logstash interface so perfectly. It doesn’t give me everything I’d get with Splunk, but I’ve just touched the functionality I can extract with Logstash,” as Lhotsky puts it.

Finally, he suggests the popular Graphite for data visualization and graphing all the log data you’ve now collected.

As Lhotsky says, this just how he tried to match Splunk-like functionality with open source tools, and it’s still a work in progress.

Below is a video from the CEO of Splunk explaining why his product is unchallenged in the space, and Just What exactly Splunk is.


“Why Splunk?”

Godfrey Sullivan, Chairman and CEO of Splunk, gives you the essential overview of Splunk. Your machine data contains a definitive record of all user transactions, customer behavior, machine behavior, security threats, system health, fraudulent activity and more. Splunk can help you take this machine data and make business sense of it. We call this operational intelligence. Learn how Splunk can help turn silos of machine data into actionable insights for IT and the business.

ROI for Big Data and Analytics

I had the honor to be on The Interop Conference Big Data Panel in Las Vegas yesterday.  The panel was composed of friends from Cloudera, Datameer, Aryaka and it was part of an all-day workshop led by Big Data celebrity and evangelist Chris Taylor.  The focus of the discussion was “The Future of Data.”  The audience was composed of very savvy technical leaders from diverse industries from financial services to retail to universities.

Debates like these can sometimes derail into sales pitches and friendly remarks.  This time, though, the discussion, brilliantly orchestrated by Matt Marshall, Founder and CEO of VentureBeat, turned into a very passionate exchange on the key themes challenging our industry.  We got so excited that Matt and Chris let us go over time to engage with the audience.
By the time we were done, I realized we hadn’t touched on a key theme Matt was interested in: Big Data and ROI.  Many of you will be asked to justify the investment you’re making in Big Data Analytics technology.  ROI is key term you’ll hear.  It stands for “Return On Investment”.

Sure, there are many ways to justify technology investments and firms like Gartner and Forrester have built such models.
I’ll tell you this though.  Companies that are trying to look at Data Analytics as a tactical budget item are in trouble.  Think about it this way: do you have to justify the return on investment of your financial department?  Probably not.  Why?  Because you need it to better run your business, safeguard yourself from exposure and spot opportunities before it’s too late – in short, to run your business better.

The same goes of Big Data and Analytics.  Data Analytics will have immediate and long term return on investment on your culture, your processes and your bottom line.  Now, you do want to use the most appropriate technology so you can avoid burning money on the wrong things…but that’s a different question.  We happen to believe we have the most effective option for Terabyte-range Data Analytics problems.

If you are still running into ROI discussion issues – try this tactic: figure out the cost of a “wrong decision”.  Meaning – what happens when your company, your executive team and/or your front-line employees execute the wrong moves because they didn’t have the right insights?

A customer of ours recently evaluated that the wrong “first move” could cost $50,000 in straight cost or lost opportunity.  And that’s the first move.  Unfortunately, “wrong first moves” rarely happen in isolation and the bill can quickly increase in an uncontrollable manner.  How is that for an ROI?

-Source: B.Runor