Using NeoDatis OODB for unit testing

February 15, 2009 by nailsandhammers

Related to the previous blog entry about using db4o for unit testing – I have tried an alternative framework NeoDatis, which is a small and simple OODB for Java, Groovy and .NET. The main reason why I started looking for alternatives was the fact that the DB4O GUI tool Object Manager was only available as part of the commercial license and NeoDatis did have simple GUI object browser. That’s not an issue anymore – DB4O OME is available now. The GPL license was another reason. From my perspective and because of the way I am using this framework for unit testing and temporary storage, the LGPL license just makes more sense than the full GPL that DB4O uses.

The migration from DB4O to NeoDatis was very straightforward, the only difference I had to figure out was on querying objects by class:

//db4o version:
ObjectContainer db = Db4o.openFile("CARS.DAT");
ObjectSet results = db.queryByExample(Car.class);
//returns also objects Cabrio, Sedan, Hatchback, ...

//NeoDatis version
ODB odb = ODBFactory.open("CARS.DAT");
CriteriaQuery query = new CriteriaQuery(Car.class);
query.setPolymorphic(true);	//in order to return the subclasses of Car
Objects results = odb.getObjects(query);
//because simple odb.getObjects(Car.class) doesn't return the subclasses of Car,
//it needs to be explicitly set as polymorphic query.

There are probably more differences than that, and even deeper ones when it comes to features, robustness, reliability, support, but I am just trying to find the right tool for my work, not one hammer for everything.
Speaking of which, I am getting more and more convinced that we might be rushing to use the relational databases a bit too much, even when it is not necessary or not really appropriate. The OODBs for storing objects is one alternative option, Amazon SimpleDB, Apache Hadoop, CouchDB or Google’s BigTable are other data storage options with different benefits and applicability.

Simple configurable event processing application with Esper and Spring

January 15, 2009 by nailsandhammers

As I was working on the CEP application I wanted to be able to add/remove the Esper statements, or change the expression text without having to recompile, or restart the process.  I used the Spring xml configuration file and XmlBeanFactory to define the eventProcessor bean and the statements that it will be executing against the event stream.

Let’s imagine a hypothetical system that monitors a building, which generates an event (ZoneEnteredEvent, ZoneAbandonedEvent) whenever a person enters or leaves a zone (by swiping a card, RFID, face recognition, whatever). The eventProcessor definition looks like this:

<bean id="eventProcessor" class="com.nailsandhammers.espring.processor.EventProcessor">
<property name="statements">
	<list>
			<ref bean="tooMuchExposureStatement" />
			<ref bean="duplicateEntryStatement" />
		</list>
	</property>
</bean>

When the eventProcessor bean is instantiated by the application it also instantiates all objects defined in the statements list.

The statements are then defined like this:

<bean id="tooMuchExposureStatement"
class="com.nailsandhammers.espring.statement.TooMuchExposureStatement">
<property name="expressionText"
value="select * from pattern[every s=ZoneEnteredEvent -> (timer:interval(30 minutes) and not f=ZoneAbandonedEvent(s.subjectId=f.subjectId and s.zoneId=f.zoneId and s.zoneId='A'))]" />
</bean>
<bean id="duplicateEntryStatement"
class="com.nailsandhammers.espring.statement.DuplicateEntryStatement">
<property name="expressionText"
value="select * from pattern[every s=ZoneEnteredEvent -> (f=ZoneEnteredEvent(s.subjectId=f.subjectId and s.zoneId!=f.zoneId)) and not a=ZoneAbandonedEvent(s.subjectId=a.subjectId and s.zoneId=a.zoneId)]" />
</bean>

The first statement tooMuchExposureStatement ensures that nobody stays in the Zone A longer than 30 minutes (let’s say it’s dangerous there).
The second statement duplicateEntryStatement controls that somebody entered a Zone, they cannot enter another Zone without abandoning the current one. Otherwise the person is clearly in two places at the same time, or it means somebody’s identity has been compromised, or the system is not working correctly. In any case, it is something to be looked at.

The statement could be any object that implements com.espertech.esper.client.UpdateListener and of course before the eventProcessor starts accepting any kind of events, it needs to make sure the statements get registered with the Esper CEP engine. In order to make the statements expressions configurable as well I created an abstract superclass EsperStatement that contains the expressionText and the register method that adds the statement to the Esper engine:


public abstract class EsperStatement implements UpdateListener {
    private String expressionText;

    public EsperStatement() {
        super();
    }

    public void register(EPServiceProvider serviceProvider) {
        EPStatement epStmt = serviceProvider.getEPAdministrator().createEPL(
                expressionText);
        epStmt.addListener(this);
    }

...

}

When eventProcessor starts up, it needs to initialize the Esper engine (see the documentation and samples) and register the statements:

public class EventProcessor {
...
	EPServiceProvider serviceProvider;
	private void registerEsperStatements() {
		if (statements == null || statements.size() == 0)
			return;
		serviceProvider = initServiceProvider();
		for (int i = 0; i < statements.size(); i++) {
			EsperStatement stmt = (EsperStatement) statements.get(i);
			stmt.register(serviceProvider);
		}
	}
...
}

So in the end the statements and the expressions that are used to process events are configurable components, that can be added, removed and changed. With a little bit more coding it would be possible to force the eventProcessor to reload the config file and reinitialize the CEP engine with the changed statements without even restarting the application process.
Happy experimenting with CEP and Esper!

Using db4o for unit test results verification

December 30, 2008 by nailsandhammers

There was a situation when I needed to verify status of an object that has been supposedly created by the component I was testing, but I couldn’t reach it from the TestCase. It was not being returned by the method, nor it was persisted in any way I could easily use. It existed only for a fraction of a second and then it was sent to another component/class. This is pretty common when you have a system that is doing any kind of remote calls or asynchronous messaging. In my case I was working on an CEP application.

Of course there are several refactoring solutions that might help,  but Gerard Meszaros in his xUnit Test Patterns describes another way how to deal with this situation – a TestSpy pattern. TestSpy is a class that I can ‘inject’ to the tested class in order to instrument and capture its behaviour. I used the TestSpy to persist objects that were being created in the tested method so I can verify them later.

I had wide number of choices how to implement the persistence – using a file, a regular sql database, embedded database like HSQLDB or H2.  I really wanted something simple, no extra server administration, table creation or  O/R mapping definition. I wanted to store and retrieve objects, so I decided to give it a try with an embedded object-oriented database . I chose db4o for no particular reason except that it seems to be pretty well established as a product and is coming with an open source license (GPL, though).

Despite having not much of former exposure to db4o, it turned out pretty quick and simple. I followed the tutorial and I had the whole testing up and running in about 30 minutes.

Here is the ‘TestSpy’ class:

public class TestStore {
	private static String dbFileName = "DB4O.DATA";
	private ObjectContainer	db;

	public TestStore() {
		db = Db4o.openFile(dbFileName);
	}

	public void close() {
		db.close();
	}

	// store object to the database
	public void store(Object dataObj) {
		db.store(dataObj);
	}

	// retrieve stored objects using an example object
	public List queryByExample(Object example) {
		ObjectSet resultSet = db.queryByExample(example);
		return resultSet;
	}

	// retrieve stored objects of the same class
	public List queryByClass(Class clazz) {
		ObjectSet resultSet = db.queryByExample(clazz);
		return resultSet;
	}
}

That’s it, that’s all the db4o-specific code I needed.

And here is the tested method:

public class EmailSender {

	// send emails to all recipients using the NewYearGreeting email template
	public void sendNewYearGreetings(String[] recipients) {
		for(int i=0; i<recipients.length; i++) {
			NewYearGreetingEmail email = new NewYearGreetingEmail(recipients[i]);
			smtpServer.send(email);

			//testability support
			if(AppConfig.ALLOW_TEST_SPY) {
				testSpy.store(email);
			}
		}
	}

	private TestStore testSpy;
	//set the TestSpy
	public void setTestSpy(TestStore testStore) {
		testSpy = testStore;
	}
}

Finally, here the testing and verification code from the TestCase:

public void testSendNewYearGreetings() {
	try {
		String[] recipients = new String[]{"joe's_addr", "bob's_addr",
								"ann's_addr"};
		TestStore testStore = new TestStore();
		EmailSender emailSender = new EmailSender();
		emailSender.setTestSpy(testStore);
		emailSender.sendNewYearGreetings(recipients);

		List resultSet = testStore.queryByClass(NewYearGreetingEmail.class);
		if(resultSet.size()!=3)
			fail("Unexpected number NewYearGreetingEmail objects!");
		//TODO - add further verification
	}
	catch(Exception ex) {
		fail("testSendNewYearGreetings Failed");
	}
}

Of course, this testing method changes the tested code somewhat. As usual in this world, there is no free lunch and adding more testability to the code introduces some features that may need to be removed in production. If this is an issue, then you can use some of the ‘conditional compilation’ techniques :

  • use ant to remove the test code from the production code as described for example here.
  • you can even use the gcc preprocessor to have ‘real’ conditional compilation as described here.
  • or you can use a static final variable that drives the execution of the test logic. This feature takes the advantage of the fact that the compilers do not generate unreachable code, so in fact your class would not even contain calls to the Test Spy methods. See the AppConfig.ALLOW_TEST_SPY variable being used in the code above.

In this case the OODB did exactly what I needed – allows me to store and retrieve my objects without too much dancing around. Pretty clean and easy to set up. I will see if this is the right hammer for the testing nail.

Release It!

December 19, 2008 by nailsandhammers

As I mentioned in the previous blog entry, Michael Nygard’s Release It! gives some advices how to deploy and run systems. But the most important point is that is changes the way people think about system development. The gap between developing a piece of code that runs in a lab and having a complete system that can run for days (weeks, months, years) without glitches and can sustain all kinds of extreme and often unpredictable conditions is huge. The more important and critical the system is, the more difficult and costly it is to ensure those reliability and availability parameters.

The book shows some interesting what-could-go-wrong scenarios,  and outlines some solutions and patterns how to address them, but the main value is that it illustrates how apparently unrelated things could just come together and bring the whole system down.  Check the recent Attack of Self-Denial post on Michael Nygard’s blog.

Toronto Stock Exchange down

December 17, 2008 by nailsandhammers

Not a very good day today. The Toronto Stock Exchange (TSX) was down the whole day because of some data feed problem, supposedly. This is happening not long after the London Stock Exchange (LSE) was unavailable for most of the day due to a system failure.  And the list goes on and on – Blackberry outage, Amazon, Twitter, Google, Apple’s MobileMe  …

Some of these system failures cause significant damage to company reputation, or a hard economical impact, but it makes me think how much of other software is running out there, controlling nuclear plants, medical care systems, military systems, air-traffic control, government. These systems are still developed by people and what I am asking is – do they have some kind of a secret recipe how to create reliable, available and secure systems? If yes, why not to share it with others? And if not, then maybe there is a reason to be scared!

Of course there isn’t a simple guideline how to ‘create a system that never fails’, but there certainly are practices that lead to better, more reliable software that at least fails less often. To get started, I would certainly recommend to read Michael Nygard’s Release It book. Check out the next blog entry!

Welcome to Nails and Hammers!

December 17, 2008 by nailsandhammers

This blog is mostly about software engineering and related technologies. That’s what I do, even enjoy doing – creating solutions in Java, JEE, Grails, but also .NET, C#, Python, you name it. I am also interested in CEP, BAM, and the whole mystery of why the software systems still suck even though so many smart people work on technologies that are supposed to make the application development ‘easier, faster and less buggy’.

Maybe it’s because at the end of the day it’s the people that get to use those technologies like tools to build things, and as people say: If the only tool you have is a hammer, the next thing you are going to hit might be your fingernail!