Welcome!

Apache Authors: Pat Romanski, Liz McMillan, Elizabeth White, Christopher Harrold, Janakiram MSV

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog, Apache, FinTech Journal

@DevOpsSummit: Blog Post

DevOps and SQL Review By @Datical | @DevOpsSummit [#DevOps]

DevOps patterns are in the constant crusade to bring high-quality products to market faster

Automating SQL Review to Save Time and Money

I’ve spent the majority of my tech career in startups.  I love the fast pace, the opportunity to learn new things, and the sense of accomplishment that comes from bringing a successful new product to market.  I began my career in Quality Assurance.  In startups, you rarely enjoy the low ratio of Developers to QA Engineers that you might in a large enterprise.  As a QA engineer in a startup, your inbox is always much more full than your outbox. You are the last gate before the next release so you’re always under the microscope. In an early stage startup you are most likely also the “Customer Support” team, so when an issue is hit in production you become VERY popular.

As someone in that position, I always kept an eye out for the right tools to lighten my load without sacrificing any of my own personal quality standards for the work I was doing.  This is how I came across FindBugs about 10 years ago.  The first time I ran it and shared the output with the development engineers on my team they felt that the tool emitted more false positives or “nitpicky” patterns than true bugs.  But over time, as we tweaked and extended the checks performed to cover our specific needs and correlated the data from FindBugs with actual counts of bugs found in test and production, FindBugs became an integral part of our nightly and on-demand builds.  The reports were an excellent early indicator of potential issues and allowed developers to rectify misdeeds before we used up testing cycles or troubleshooting time in operations.  The developers on my team also started committing fewer and fewer infractions as the daily reminders they got from our build system helped them to change their bad habits into safer, better performing, more stable code. Release cycles shortened, product quality improved, and customer satisfaction rose proving that an ounce of prevention really is worth a pound of cure.

As Enterprise IT embraces agile development practices and adopts DevOps patterns in the constant crusade to bring high-quality products to market faster, DBAs are really starting to feel the pinch.  The description above of a QA Engineer in a software startup is apt.  With more frequent releases the DBA’s inbox of SQL scripts to write, review, modify or optimize is always more full than her outbox. The DBA is the last bastion of defense for data quality, data security, and data platform performance and is therefore under constant scrutiny. When there is a production outage, the DBA is among the first called to respond.

One of the most time consuming tasks for the Fortune 50 DBAs we work with is SQL review.  Some DBAs are allocating 70% of their time manually reviewing SQL scripts.  They are checking for the same things in SQL that tools like FindBugs are looking for in Java code: code patterns that indicate logical problems, security flaws, performance issues, and non-compliance to internally defined best practices or externally mandated regulations.

It’s clear that DBA’s need a tool that does for them what FindBugs did for my team a decade ago.  Static analysis for SQL is nothing new, but current offerings only go so far.  Typically, they evaluate the SQL statements with no contextual sensitivity. This omission severely limits the productivity and quality gains that can be achieved because so much of Database Lifecycle Management is being aware of Who is doing What, Where and When.  For example, an organization may allow privilege grants and INSERT statements in a TEST environment, but never allows such activity in an automated session in PROD. Any static analysis tool for SQL must take environmental parameters into consideration.

Also complicating matters is the nature of database ‘versioning.’  While your application is packaged, versioned and replaced wholesale from release to release, the database schema that supports your application is persistent and evolves over time.  What’s more, external compliance standards (SOX or PCI DSS for example) and internal audit requirements often dictate that incremental changes to the database be rigidly controlled and tracked in a well-defined process. This means the DBA must also confirm (through manual process and reviewing SQL for the appropriate comments) that the change can be traced to its cause and the application of the change can be traced through each environment.

The Datical DB Rules Engine was designed and implemented to satisfy the unique set of challenges posed by SQL review & static analysis.  Here are just a few of the reasons that Datical DB enables acceleration through static analysis safely and sanely.

  • Models Make for Powerful Evaluation – Datical DB abstracts the application schema into a strictly defined and validated object model. Authoring powerful rules is fast, straightforward and simple. Once they are written they are enforced every time a Forecast or Deploy is performed on any database in the lifecycle.
  • Environmentally Aware Change Validation - The model includes information about the client environment and various database instances in your applications lifecycle. Your rules can be written to allow maximum flexibility in early stage environments and maximum security in sensitive environments simultaneously.
  • Easily Confirm Internal & External Audit Requirements – In Datical DB, everything you need to remain in compliance with external and internal audit requirements is tied tightly to individual changes in the Data Model.  Manual review to confirm auditability of change is replaced with automated checks that are executed every time you (or your automation frameworks) Forecast or Deploy.
  • Automatically Validate What’s Important to YOU - Provides the capability to customize analysis to cover internal best practices like naming conventions, SQL DOs and DON’Ts, and object dependency management
  • Automate The Boring Stuff. Get Back To The Fun Stuff - Like many static analysis tools for code, Datical DB integrates into your build and deployment systems in a few mouse clicks. Now every time you build or promote an application, Rules validations are performed and a report is generated for dissemination throughout the organization. Your DBAs, having considerably reduced the time they spent with eyes on the screen reading SQL, are concentrating on more strategic projects and problems.
  • Better Coding Means Fewer Bugs - DBAs author rules and share them with development.  Development then has a codified repository of what is and is not acceptable in their organization to work against. Fewer bugs escaping DEV saves time and money.
  • Increasing Operations Involvement In Database Development – The Rules Engine is tightly integrated with Datical DB Forecast.  This feature allows you to simulate database change without actually altering the target database.  When DBAs share their Rules with Operations, Operations can run nightly Forecasts against STAGE or PROD to ensure that what’s currently in DEV or TEST will comply with the stricter validations performed downstream, once again finding problems earlier in the lifecycle when they are cheaper and easier to fix.

More Stories By Pete Pickerill

Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. Pete has spent the majority of his career in successful startups and the companies that acquired them including Loop One (acquired by NeoPost Solutions), WholeSecurity (acquired by Symantec, Inc.) and Phurnace Software.

@ThingsExpo Stories
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Founded in 2000, Chetu Inc. is a global provider of customized software development solutions and IT staff augmentation services for software technology providers. By providing clients with unparalleled niche technology expertise and industry experience, Chetu has become the premiere long-term, back-end software development partner for start-ups, SMBs, and Fortune 500 companies. Chetu is headquartered in Plantation, Florida, with thirteen offices throughout the U.S. and abroad.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
I think DevOps is now a rambunctious teenager - it's starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Major trends and emerging technologies – from virtual reality and IoT, to Big Data and algorithms – are helping organizations innovate in the digital era. However, to create real business value, IT must think beyond the ‘what’ of digital transformation to the ‘how’ to harness emerging trends, innovation and disruption. Architecture is the key that underpins and ties all these efforts together. In the digital age, it’s important to invest in architecture, extend the enterprise footprint to the cl...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessio...
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
No hype cycles or predictions of zillions of things here. IoT is big. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, Associate Partner at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He discussed the evaluation of communication standards and IoT messaging protocols, data analytics considerations, edge-to-cloud tec...
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | CloudEXPO New York event will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
CloudEXPO | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
The best way to leverage your CloudEXPO | DXWorldEXPO presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering CloudEXPO | DXWorldEXPO will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at CloudEXPO. Product announcements during our show provide your company with the most reach through our targeted audienc...
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.