Welcome!

Apache Authors: Pat Romanski, Liz McMillan, Elizabeth White, Christopher Harrold, Janakiram MSV

Related Topics: Apache

Apache: Blog Post

Apache Hadoop: Technical Debt Decreased by 14% Through Code Refactoring

Initial Technical Debt of the project reduced from 136 to 117 days of remediation

Technical Debt is worth nothing if no pragmatic action is taken into code, in order to control and tackle it. To illustrate Scertify's capability to automatically correct code defects that increase this unintended debt, we performed code refactoring on two subprojects of the Hadoop project : Hadoop Common and Hadoop Mapreduce. Thanks to Scertify, we were able to correct 25K defects in 2 minutes. In other words, 14% of the Technical Debt has been written-off without any human effort needed.

Initial analysis
According to Wikipedia, Apache Hadoop is "an open-source software framework that supports data-intensive distributed applications". This framework contains several projects, Common and Mapreduce are two important ones with respectively 120K and 162K lines of code (blank lines and comments excluded). The version we worked with is the last development version : 3.0.0-SNAPSHOT. We ran Scertify Refactoring Assessment, our open-source plugin for Sonar, on the projects, in order to get an overview of their technical debt. Technical debt is defined as the amount of time needed to correct all defects detected. As you can see on screen-shots below, Common has a technical debt of 70 days and Mapreduce of 66 days. Scertify Refactoring Assessment also computes the potential of automatic correction of the technical debt : the debt write-off. They both have a good potential for automatic refactoring, respectively 38 and 36 days. So, the next step is to use Scertify to perform this automatic refactoring. By the way, if you would like to try it with your own source code, a trial version of Scertify is available here.

Hadoop Common Original Techdebt

Hadoop Mapreduce Original Technical debt

We scrolled among the various errors and we chose 8 rules to perform the demonstration.

Refactoring rules for the demonstration

Here's a presentation of the refactoring rules we used in this demonstration. As you can see, some rules need parameters to be efficient. This is the case of rules regarding logging. The logging framework used in those project is Apache Common logging, so we configured the rules to use this framework.

AvoidPrintStackTrace

This rule reports a violation when it finds a code that catch an expression and print its stack trace to the standard error output. A logging framework should be used instead, in order to improve application's maintainability. The refactoring replace a call to print stack trace by a call to a logging framework. The rule can also declare the logger in the class and make the required imports. Here's an example of the original code and the refactored code in the class GenericWritable.

Original code:

catch (Exception e) {
      e.printStackTrace();
      throw new IOException("Cannot initialize the class: " + clazz);
}

Refactored code:

catch (final Exception e) {
      LOG.error(e.getMessage(), e);
      throw new IOException("Cannot initialize the class: " + clazz);
}

In this case, LOG was not declared so it was added to the class and import were made :

private static final Log LOG = LogFactory.getLog(GenericWritable.class);

InefficientConstructorCall
Calling the constructor of a wrapper type, like Integer, to convert a primitive type is a bad practice. It is less efficient than calling the static method valueOf.

PositionLiteralsFirstInComparisonsRefactor

This rule checks that literals are in the first position in comparisons. The refactoring invert the literal and the variable. This ensures that the code cannot crash due to the variable being a null pointer.

AddEmptyStringToConvert

Using the concatenation of an empty string to convert a primitive type to a String is a bad practice. First of all, it makes the code less readable. It is also less efficient in most cases (the only case where the string concatenation is slightly better is when the primitive is final). Here's an example taken from class MD5MD5CRC32FileChecksum.

Original code:

xml.attribute("bytesPerCRC", "" + that.bytesPerCRC);

Refactored code:

xml.attribute("bytesPerCRC", String.valueOf(that.bytesPerCRC));

GuardDebugLogging
When a concatenation of String is performed inside a debug log, one should check if debug is enabled before making the call. Otherwise, the String concatenation will always be done. The refactoring adds a guard before the call to debug. In this case, it is configured to use the method isDebugEnabled(), since we use Apache's log. Below is an example of refactored code taken from class ActiveStandByElector:

if(LOG.isDebugEnabled()){
        LOG.debug("StatNode result: " + rc + " for path: " + path + " connectionState: " + zkConnectionState + " for " + this);
}

IfElseStmtsMustUseBraces

This rule finds if statements that don't use braces. The refactoring adds required braces.

UseCollectionIsEmpty

This rule finds usage of Collection's size method to check if a collection is empty. Rather than using size(), it is better to use isEmpty() making the code easier to read. The refactoring replace comparisons between size and 0 with a call to isEmpty().

LocalVariableCouldBeFinal

This method flags local variables that could be declared final and are not. The use of the final keyword is a useful information for future code readers. The refactoring adds the "final" keword. This is not a critical rule, but since it has a huge number of violations, it is useful to get rid of them quickly with automatic refactoring.

Scertify's refactoring results
So we ran Scertify on both projects to detect and refactor those rules. On each project it took around 1 minute to perform the full process. Scertify generates an html report with information on errors detected and corrected. Below is a summary of all errors corrected in the two projects. Many minor things were corrected, but also more important ones. Overall, it took 2 minutes to correct 25392 defects. Not so bad isn't it? Those defects include both minor violations and more critical violations in term of maintainability, performance or robustness.

Violations refactored

As you can see on screen-shots below, with those defects corrected the technical debt of each project has been reduced of 10 days. Overall, that's 20 day of technical debt that have been written-off.

Refactored Common technical debt

Refactored Mapreduce technical debt

Last but not least, Hadoop contains many unit tests and of course we made sure that they still succeed after the refactoring. To conclude, thanks to Scertify's refactoring features we were able to efficiently correct 25K defects in few minutes. We are glad to make the refactored code available to community, you can download it below. We will continue to do such refactoring on open-source applications, so if you have an idea for an open-source project that could leverage such refactoring, just let us know!

Download the source files

More Stories By Michael Muller

Michael Muller, a Marketing Manager at Tocea, has 10+ years of experience as a Marketing and Communication Manager. He specializes in technology and innovative companies. He is executive editor at http://dsisionnel.com, a French IT magazine and the creator of http://d8p.it, a cool URL shortener. Dad of two kids.

@ThingsExpo Stories
The age of Digital Disruption is evolving into the next era – Digital Cohesion, an age in which applications securely self-assemble and deliver predictive services that continuously adapt to user behavior. Information from devices, sensors and applications around us will drive services seamlessly across mobile and fixed devices/infrastructure. This evolution is happening now in software defined services and secure networking. Four key drivers – Performance, Economics, Interoperability and Trust ...
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software in the hope of capturing value in IoT. Although IoT is relatively new in the market, it has already gone through many promotional terms such as IoE, IoX, SDX, Edge/Fog, Mist Compute, etc. Ultimately, irrespective of the name, it is about deriving value from independent software assets participating in an ecosystem as one comprehensive solution.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
SYS-CON Events announced today that Juniper Networks (NYSE: JNPR), an industry leader in automated, scalable and secure networks, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Juniper Networks challenges the status quo with products, solutions and services that transform the economics of networking. The company co-innovates with customers and partners to deliver automated, scalable and secure network...
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
SYS-CON Events announced today that Hitachi, the leading provider the Internet of Things and Digital Transformation, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Hitachi Data Systems, a wholly owned subsidiary of Hitachi, Ltd., offers an integrated portfolio of services and solutions that enable digital transformation through enhanced data management, governance, mobility and analytics. We help globa...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/...
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market across the USA and Europe, we work with a variety of customers from emerging startups to Fortune 1000 companies.
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Analytic. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
In his keynote at @ThingsExpo, Chris Matthieu, Director of IoT Engineering at Citrix and co-founder and CTO of Octoblu, focused on building an IoT platform and company. He provided a behind-the-scenes look at Octoblu’s platform, business, and pivots along the way (including the Citrix acquisition of Octoblu).