Scanning your Java App with jQAssistant and Neo4j

Michael Hunger’s talk had an elegant solution to an actual, unsolved problem of mine

After attending a software development conference, you usually return back home with a bunch of new ideas and cool things you want to try out. But most of the time your ideas remain in experimental stage, since it’s often hard to find a problem that’s just right for all newfound solutions…

But returning from GeekOut 2014 was different – Michael Hunger’s talk “Using Ascii Art to Analyze Your Source Code with Neo4j and OSS Tools” had an elegant solution to an actual, unsolved problem of mine that had surfaced just a couple of weeks earlier.

 

Challenge

A system I am involved in as a developer relies on custom parsing of the source code to find all concrete subclasses of a particular class, excluding inner classes. These subclasses represent units for which user access can be configured. If this list is incomplete, the configuration will be incomplete, which in turn means missing user access. We had just seen our first case of generics showing up in this class hierarchy, and the parser couldn’t handle that properly.

Solution

The objective is to extract all concrete subclasses of a particular class into a custom format file, and this should be part of the build. This file serves as a template for access control configuration for the scanned rich client application.

If you have a Maven project, this can be done quite easily with jQAssistant and Neo4j. Here are the basic steps:

  1. Setup jQAssistant
  2. Build your Maven project
  3. Scan your compiled classes with jQAssistant to create a Neo4j representation of your code
  4. Query the database to extract the information you need
  5. Generate the result

The steps will be described in more detail below, using this sample project. It’s a real-world example, but the actual code scanned and the custom report plugin have been replaced with sample code.

Step 1 – Setting Up jQAssistant

To set up jQAssistant, make sure you have the following:

  • JDK 7
  • Maven 3 and access to Maven Central Repository
  • A Maven project

Add the following to your settings.xml, located in the .m2 folder of your home directory:

Mattias Olofsson

Senior Consultant
Mattias is a software developer with project management experience. He is a quick learner with excellent analytical, design and problem solving skills. He has long experience from design and development of technically challenging systems. He has also been involved in team and technical project management as well as architecture work. He is comfortable working independently or as part of a team and always strives to perform his best and deliver on time and in full.
<pluginGroups>
    <pluginGroup>com.buschmais.jqassistant.maven</pluginGroup>
</pluginGroups>

This makes it easier to run jQAssistant from the command line. Now you’re good to go.

Step 2 – Build Your Maven Project

Since jQAssistant scans bytecode, you need to build your Maven project.

mvn clean install

Step 3 – Scan Your Compiled Classes

Before you can scan your application, you need to add the jQAssistant Maven plugin to your POM:

<properties>
   <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
   <jqassistant.version>1.0.0-M3</jqassistant.version>
</properties>


<build>
   <plugins>
      <plugin>
         <groupId>com.buschmais.jqassistant.scm</groupId>
         <artifactId>jqassistant-maven-plugin</artifactId>
         <version>${jqassistant.version}</version>
         <configuration>
            <reportProperties>
               <report.file>result.txt</report.file>
            </reportProperties>
         </configuration>
         <dependencies>
            <dependency>
               <groupId>com.buschmais.jqassistant.plugin</groupId>
               <artifactId>jqassistant.plugin.java</artifactId>
               <version>${jqassistant.version}</version>
            </dependency>
            <dependency>
               <groupId>com.mattias</groupId>
               <artifactId>report-writer</artifactId>
               <version>1.0-SNAPSHOT</version>
            </dependency>
         </dependencies>
      </plugin>
   </plugins>
</build>

This also includes configuration of the report writer described later, the file to which the result should be written.

Now you can scan your code using

mvn jqassistant:scan

Step 4 – Query the Neo4j Database

With your code scanned, it’s time to write your Cypher query. The Neo4j web interface is very useful for experimenting with queries, start it by executing

mvn jqassistant:server

Now you can browse to https://localhost:7474 and start writing and testing your queries.

Neo4j web interface

First, let’s locate the base class that we want to find concrete subclasses for. In the sample code we have a hierarchy of animals, with the base class Animal. So, to find that class, we can use the following query:

match
  (t:Type)
where
  t.name = 'Animal'
return
  t as Type

Executing that query in the web interface will return one node, the Animal class:

Now, if we extend the query a little bit we can get all subclasses instead.

match 
  (t:Type)<-[:EXTENDS*]-(sc:Type)
where 
  t.name = 'Animal'
return 
  sc as Type

Clicking the nodes will reveal that we now have found some animals, but also some abstract classes such as Fish and Insect. To get rid of those, we enhance the query:

match 
  (t:Type)<-[:EXTENDS*]-(sc:Type)
where 
  t.name = 'Animal'
  and sc.abstract is null
return 
  sc as Type

Now we only have concrete subclasses left. The final thing to get rid of is the inner class declared by GoblinShark.

match 
  (t:Type)<-[:EXTENDS*]-(sc:Type)
where 
  t.name = 'Animal'
  and sc.abstract is null
  and not sc:Inner
return 
  sc as Type

Now only we have a query that returns all concrete subclasses of Animal, inner classes excluded.

To stop the server, return to the console from where it was started and hit Enter.

Now it’s time to create a rule file to integrate the query into the build. Rule files are XML files located in the jqassistant folder.

<jqa:jqassistant-rules xmlns:jqa="https://www3.buschmais.com/jqassistant/core/analysis/rules/schema/v1.0">

    <concept id="my-rules:ExtractSubClassesOfAnimal">
        <requiresConcept refId="java:InnerType" />
        <description>Extracts subclasses of Animal</description>
        <cypher><![CDATA[
		  match
            (t:Type)<-[:EXTENDS*]-(sc:Type)
          where
            t.name = 'Animal'
            and sc.abstract is null
            and not sc:Inner
          return
            sc as Type
        ]]></cypher>
    </concept>

    <group id="default">
        <includeConcept refId="my-rules:ExtractSubClassesOfAnimal"/>
    </group>

</jqa:jqassistant-rules>

Step 5 – Generate the Result

Now that we have a query that do the extraction from the compiled code, we just need to put that in a file. In this example, we just want to write the fully qualified name for each class found by the query. To do that, we implement a custom report writer:

package com.mattias;

import com.buschmais.jqassistant.core.analysis.api.Result;
import com.buschmais.jqassistant.core.analysis.api.rule.Concept;
import com.buschmais.jqassistant.core.analysis.api.rule.Constraint;
import com.buschmais.jqassistant.core.analysis.api.rule.Group;
import com.buschmais.jqassistant.core.analysis.api.rule.Rule;
import com.buschmais.jqassistant.core.report.api.ReportException;
import com.buschmais.jqassistant.core.report.api.ReportPlugin;
import com.buschmais.jqassistant.plugin.java.api.model.TypeDescriptor;

import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.util.List;
import java.util.Map;

/**
 * Created by mattias on 2014-06-30.
 */
public class ReportWriter implements ReportPlugin {

    private String reportFile;

    @Override
    public void initialize() throws ReportException { }

    @Override
    public void configure(Map<String, Object> map) throws ReportException {
        this.reportFile = (String) map.get("report.file");
    }

    @Override
    public void begin() throws ReportException { }

    @Override
    public void end() throws ReportException { }

    @Override
    public void beginConcept(Concept concept) throws ReportException { }

    @Override
    public void endConcept() throws ReportException { }

    @Override
    public void beginGroup(Group group) throws ReportException { }

    @Override
    public void endGroup() throws ReportException { }

    @Override
    public void beginConstraint(Constraint constraint) throws ReportException { }

    @Override
    public void endConstraint() throws ReportException { }

    @Override
    public void setResult(Result<? extends ExecutableRule> result) throws ReportException {
        Rule rule = result.getRule();
        if ("my-rules:ExtractSubClassesOfAnimal".equals(rule.getId())) {
            try {
                BufferedWriter writer = new BufferedWriter(new FileWriter(reportFile));
                List<Map<String, Object>> rows = result.getRows();
                for (Map<String, Object> row : rows) {
                    TypeDescriptor type = (TypeDescriptor) row.get("Type");
                    writer.write(type.getFullQualifiedName() + "\n");
                }
                writer.flush();
                writer.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

When we plug in this report writer, the result from the query will be available in the Result parameter passed to the setResult method. The rows can be retrieved and written to a file in the desired format.

To let jQAssistant know about your plugin, create a file called jqassistant-plugin.xml and place it in the META-INF folder of your report writer project.

<jqa-plugin:jqassistant-plugin xmlns:jqa-plugin="https://www3.buschmais.com/jqassistant/core/plugin/schema/v1.0" name="Custom Report">
    <description>Example plugin defining a custom report.</description>
    <report>
        <class>com.mattias.ReportWriter</class>
    </report>
</jqa-plugin:jqassistant-plugin>

The final step is to let jQAssistant execute your query and generate the report, which is done by executing

mvn jqassistant:analyze

As simple as that.

Thanks to Michael Hunger for a great talk on jQAssistant and Neo4j at GeekOut 2014, and many thanks to Dirk Mahler for helping out, sharing knowledge and enhancing jQAssistant to allow plugging in your own custom report writer. Awesome.

For more details check out https://jqassistant.org where you’ll also find reference documentation and examples.

The refactoring safety net

Your black box tests will give you the confidence and keep you safe

Don’t underestimate the value of keeping your unit tests in shape. When it’s time for refactoring, your black box tests will help you get the job done faster.

Unit Test

There are many good reasons why you should unit test your code, an important one is that if done right, they serve as a safety net when you do refactoring. Since no developer writes the ideal solution on the first go, refactoring is needed all the time. But to take advantage of your tests when refactoring, your design must allow changing the implementation without rewriting your tests.

The tests that are going to help you in the process are your black box tests, the ones that test your functionality without bothering about implementation details. Any white box tests you might have, that know about the internal structures of your components, will make your mission more risky not to mention painful – and no developer likes to change more code than necessary.

The less your tests know about the internal implementation of your components, the more useful they will be during refactoring. If you have to change your unit tests just as much as the rest of the code, you can’t use them to ensure that you still have the same behavior and quality.

Enforce a good design

When you develop a new component, your tests help you enforce good design. Developers who don’t like writing tests are quite good at coming up with excuses why they shouldn’t – “this can’t be tested, is too difficult”. Well, if it’s difficult to test, it’s probably difficult to use. Maybe it can be designed a bit differently, perhaps the business logic is in the wrong place, or there’s some static dependency that shouldn’t be there.

Coding against interfaces, keeping clear dependencies and loose coupling between components are all factors that simplify your testing quite a bit, and mocking will be a lot easier. Try to separate different concerns, avoid mixing code for transaction management and database access with your business rules. Usually, this also makes it easier to put the right code in the right place.

Getting inside the box

If you feel the urge to get inside the box and test a private method, access or set private fields, your class may be responsible for more than it should, perhaps it has unclear dependencies, or it could be too tightly coupled with other components.

Have a look at the design before bringing on test frameworks with capabilities for breaking encapsulation – save that for the cases when there is no other way out, or you really need to verify implementation details. In a well designed system with unit tests protecting the interface and functionality of each component, refactoring will be a lot faster – and your black box tests will give you the confidence and keep you safe.

Mattias Olofsson

Senior Consultant
Mattias is a software developer with project management experience. He is a quick learner with excellent analytical, design and problem solving skills. He has long experience from design and development of technically challenging systems. He has also been involved in team and technical project management as well as architecture work. He is comfortable working independently or as part of a team and always strives to perform his best and deliver on time and in full.