Welcome!

Apache Authors: Liz McMillan, Carmen Gonzalez, Elizabeth White, Pat Romanski, Christopher Harrold

Related Topics: Apache

Apache: Blog Post

Apache Hadoop: Technical Debt Decreased by 14% Through Code Refactoring

Initial Technical Debt of the project reduced from 136 to 117 days of remediation

Technical Debt is worth nothing if no pragmatic action is taken into code, in order to control and tackle it. To illustrate Scertify's capability to automatically correct code defects that increase this unintended debt, we performed code refactoring on two subprojects of the Hadoop project : Hadoop Common and Hadoop Mapreduce. Thanks to Scertify, we were able to correct 25K defects in 2 minutes. In other words, 14% of the Technical Debt has been written-off without any human effort needed.

Initial analysis
According to Wikipedia, Apache Hadoop is "an open-source software framework that supports data-intensive distributed applications". This framework contains several projects, Common and Mapreduce are two important ones with respectively 120K and 162K lines of code (blank lines and comments excluded). The version we worked with is the last development version : 3.0.0-SNAPSHOT. We ran Scertify Refactoring Assessment, our open-source plugin for Sonar, on the projects, in order to get an overview of their technical debt. Technical debt is defined as the amount of time needed to correct all defects detected. As you can see on screen-shots below, Common has a technical debt of 70 days and Mapreduce of 66 days. Scertify Refactoring Assessment also computes the potential of automatic correction of the technical debt : the debt write-off. They both have a good potential for automatic refactoring, respectively 38 and 36 days. So, the next step is to use Scertify to perform this automatic refactoring. By the way, if you would like to try it with your own source code, a trial version of Scertify is available here.

Hadoop Common Original Techdebt

Hadoop Mapreduce Original Technical debt

We scrolled among the various errors and we chose 8 rules to perform the demonstration.

Refactoring rules for the demonstration

Here's a presentation of the refactoring rules we used in this demonstration. As you can see, some rules need parameters to be efficient. This is the case of rules regarding logging. The logging framework used in those project is Apache Common logging, so we configured the rules to use this framework.

AvoidPrintStackTrace

This rule reports a violation when it finds a code that catch an expression and print its stack trace to the standard error output. A logging framework should be used instead, in order to improve application's maintainability. The refactoring replace a call to print stack trace by a call to a logging framework. The rule can also declare the logger in the class and make the required imports. Here's an example of the original code and the refactored code in the class GenericWritable.

Original code:

catch (Exception e) {
      e.printStackTrace();
      throw new IOException("Cannot initialize the class: " + clazz);
}

Refactored code:

catch (final Exception e) {
      LOG.error(e.getMessage(), e);
      throw new IOException("Cannot initialize the class: " + clazz);
}

In this case, LOG was not declared so it was added to the class and import were made :

private static final Log LOG = LogFactory.getLog(GenericWritable.class);

InefficientConstructorCall
Calling the constructor of a wrapper type, like Integer, to convert a primitive type is a bad practice. It is less efficient than calling the static method valueOf.

PositionLiteralsFirstInComparisonsRefactor

This rule checks that literals are in the first position in comparisons. The refactoring invert the literal and the variable. This ensures that the code cannot crash due to the variable being a null pointer.

AddEmptyStringToConvert

Using the concatenation of an empty string to convert a primitive type to a String is a bad practice. First of all, it makes the code less readable. It is also less efficient in most cases (the only case where the string concatenation is slightly better is when the primitive is final). Here's an example taken from class MD5MD5CRC32FileChecksum.

Original code:

xml.attribute("bytesPerCRC", "" + that.bytesPerCRC);

Refactored code:

xml.attribute("bytesPerCRC", String.valueOf(that.bytesPerCRC));

GuardDebugLogging
When a concatenation of String is performed inside a debug log, one should check if debug is enabled before making the call. Otherwise, the String concatenation will always be done. The refactoring adds a guard before the call to debug. In this case, it is configured to use the method isDebugEnabled(), since we use Apache's log. Below is an example of refactored code taken from class ActiveStandByElector:

if(LOG.isDebugEnabled()){
        LOG.debug("StatNode result: " + rc + " for path: " + path + " connectionState: " + zkConnectionState + " for " + this);
}

IfElseStmtsMustUseBraces

This rule finds if statements that don't use braces. The refactoring adds required braces.

UseCollectionIsEmpty

This rule finds usage of Collection's size method to check if a collection is empty. Rather than using size(), it is better to use isEmpty() making the code easier to read. The refactoring replace comparisons between size and 0 with a call to isEmpty().

LocalVariableCouldBeFinal

This method flags local variables that could be declared final and are not. The use of the final keyword is a useful information for future code readers. The refactoring adds the "final" keword. This is not a critical rule, but since it has a huge number of violations, it is useful to get rid of them quickly with automatic refactoring.

Scertify's refactoring results
So we ran Scertify on both projects to detect and refactor those rules. On each project it took around 1 minute to perform the full process. Scertify generates an html report with information on errors detected and corrected. Below is a summary of all errors corrected in the two projects. Many minor things were corrected, but also more important ones. Overall, it took 2 minutes to correct 25392 defects. Not so bad isn't it? Those defects include both minor violations and more critical violations in term of maintainability, performance or robustness.

Violations refactored

As you can see on screen-shots below, with those defects corrected the technical debt of each project has been reduced of 10 days. Overall, that's 20 day of technical debt that have been written-off.

Refactored Common technical debt

Refactored Mapreduce technical debt

Last but not least, Hadoop contains many unit tests and of course we made sure that they still succeed after the refactoring. To conclude, thanks to Scertify's refactoring features we were able to efficiently correct 25K defects in few minutes. We are glad to make the refactored code available to community, you can download it below. We will continue to do such refactoring on open-source applications, so if you have an idea for an open-source project that could leverage such refactoring, just let us know!

Download the source files

More Stories By Michael Muller

Michael Muller, a Marketing Manager at Tocea, has 10+ years of experience as a Marketing and Communication Manager. He specializes in technology and innovative companies. He is executive editor at http://dsisionnel.com, a French IT magazine and the creator of http://d8p.it, a cool URL shortener. Dad of two kids.

IoT & Smart Cities Stories
Codete accelerates their clients growth through technological expertise and experience. Codite team works with organizations to meet the challenges that digitalization presents. Their clients include digital start-ups as well as established enterprises in the IT industry. To stay competitive in a highly innovative IT industry, strong R&D departments and bold spin-off initiatives is a must. Codete Data Science and Software Architects teams help corporate clients to stay up to date with the mod...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and Bi...
The Japan External Trade Organization (JETRO) is a non-profit organization that provides business support services to companies expanding to Japan. With the support of JETRO's dedicated staff, clients can incorporate their business; receive visa, immigration, and HR support; find dedicated office space; identify local government subsidies; get tailored market studies; and more.
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, discussed some of the security challenges of the IoT infrastructure and related how these aspects impact Smart Living. The material was delivered interac...
CloudEXPO has been the M&A capital for Cloud companies for more than a decade with memorable acquisition news stories which came out of CloudEXPO expo floor. DevOpsSUMMIT New York faculty member Greg Bledsoe shared his views on IBM's Red Hat acquisition live from NASDAQ floor. Acquisition news was announced during CloudEXPO New York which took place November 12-13, 2019 in New York City.
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
Intel is an American multinational corporation and technology company headquartered in Santa Clara, California, in the Silicon Valley. It is the world's second largest and second highest valued semiconductor chip maker based on revenue after being overtaken by Samsung, and is the inventor of the x86 series of microprocessors, the processors found in most personal computers (PCs). Intel supplies processors for computer system manufacturers such as Apple, Lenovo, HP, and Dell. Intel also manufactu...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understa...
AI and machine learning disruption for Enterprises started happening in the areas such as IT operations management (ITOPs) and Cloud management and SaaS apps. In 2019 CIOs will see disruptive solutions for Cloud & Devops, AI/ML driven IT Ops and Cloud Ops. Customers want AI-driven multi-cloud operations for monitoring, detection, prevention of disruptions. Disruptions cause revenue loss, unhappy users, impacts brand reputation etc.