Filtering Filters

Hello ! (Who needs a greeting anyway), distributed queries are a very powerful part of coherence, but can introduce show stopping performance degradation, so it is vital to get your filters right.

The filters provided by coherence, take an Extractor as a parameter, and will execute that extractor across all of the nodes, and the evaluate method will run on the result of that extractor.
This means that, if you are using a normal reflexion extractor(what is used by default when you call a filter with just a method name) new EqualsFilter(“getName”,”some_name”). This is quite slow and Inefficient,  as every object will be deserialized in order to evaluate that filter. There are two ways to avoid this,

The simplest was to avoid this is to use an Index to store those values. indexed values are kept unserialized and are matched in the same was as regular db indexes. The Filter classes provided by coherence implement IndexAwareFilter, and know how to lookup those indexes, therefore avoiding the deserialization.

The other way is to simply pass a POF extractor as a parameter, POF extractors do not deserialize an object In order to extract the desired value. However this approach is slower than having Indexes, as each object will still be evaluated, as opposed to simply looking up the Index table. However, this provides more flexibility, as you can decide at run time which fields will be used.

Custom Filters:
Sometimes you need more complex logic than the custom filters provide you, so you may decide to code your own, the simplest way to do that is to simply implement the Filter interface, however, a filter created in this manner will not use indexes even if they are relevant and will deserialize every object in that particular cache resulting in catastrophic performance.

There are a few ways to create a custom filter efficiently:

Implementing EntryFilter instead of Filter allows you to implement the evaluateEntry Method, which receives a BinaryEntry as a parameter, allowing you to extract the values yourself in a clever manner, avoiding deserializing the whole entry, however the extraction will still occur for every entry, which can be time consuming.

Extending ExtractorFilter allows you to, through its superclasse’s constructor, to take advantange of the created indexes and, by implementing the evaluateExtracted method to write your complex custom logic to evaluate that Object.

Cheers,

Server Side Versioning

Hello World!(Nothing yet on the new greeting front), I was reading a very good post about versioned put on Ben Stopford blog's, this stimulated me to run some tests and write something to perform server side versioning.
 
Versioning
As Ben explained quite well on his blog many caches will need some concept of versioning, for instance, you might want to know not only the latest version of a piece of market data for a given instrument, but you might also want the last 30, to have a view of the overall movement of that instrument.

Why Server Side Versioning
I wanted to write about server side versioning because that is the type of versioning i prefer to use, I find that Client side versioning makes distributed puts non trivial, can create bottlenecks and misses the opportunity to take advantage of Coherence’s distributed infrastructure.

The Goals

• Multiple distributed clients must be able to push data at the same time and this should not affect the versioning.
• Multiple Objects with the same business key must be kept in the cache in the same partition.
• At any given time, a client, using the well known latest version marker (-1) must be able to access the latest version of an object with a simple get.

The steps

• The first step was to create a key class (in this example called versioned key) that wraps the business key and version, and that implements KeyAssociation, using the business key as the associated key. This will ensure that all of the objects for that same business key will share a partition.
• My Model class needed a version placeholder field, this will be explained in the the map trigger step,  and is one of the main drawbacks of this approach.
• Creation of a map trigger that takes advantage of the Entries<link> getPreviousBinaryEntry() method to access the previous version of an entry, update its version and directly put it in the backing map.

Workflow

• Whenever a new VersionedKey Object is created its version is set to the well known latest version marker, in this example, -1.
• When an Entity is added for that key is added to the cache, the VersioningMapTrigger will be executed, having two different flows, one if it is the first object to be added, and another if there are already other versions of that Entity in the cache.
• If there are no other entries with that key in the cache (getPreviousBinaryEntry is null)  all you need to do is to set the version in the Entity (not its key) to 1 using a POF updater to avoid deserializing it.
• If another entry exists in the cache for that same key (getPreviousBinaryEntry is not null), several step are required
• The first step is to get the previous real version(not the version in the key, as it will be -1 since it was the latest) from the Entity that was already in the cache and set the version in the key of that Entity to the same value.
• We then set the version on the Entity that was just added to that value +1, again using a POF Updater.
• Finally we put the previous entry and previous key directly in the backing map.

Code

Tests
This approach meets all of my requirements, and to be honest there is only one thing I do not like about it, and that is having the version on my model objects, however, since I cannot easily change the value of the key that will be added after a map trigger I decided it was worth it.  I also tried some different approaches, such as my own implementation of a local cache and a backing map listener, however both of them required more backing map calls and were less performant, so I decided against it.

Unit testing POF

Hello world!(still looking for that better greeting...), It has become a recurrent topic here in the blog, ok, this is the first post and nothing is recurrent, however I hope it will be.

POF is probably the most used type of serialization for coherence, because of its performance (it is around 10-12 times faster, which saves a lot on CPU cycles and the generated Binaries are about six times smaller than in normal serialization.) is far better than normal java serialization, however POF requires some configuration and implementation, as explained in the documentation.

Another factor that need to be considered is the maintenance cost to the development process. It is important  to make sure that mappings are up-to date to avoid, at best wasting time to fix something and at worst ending up with missing data or inverting two fields.

Proposed solutions:
The easiest and crudest way to do this is to simply have a unit test that puts all of your objects to their respective caches, then gets them and compares them to assert if they are equal. This however will take some time, as you must start a cache server, and, for these tests to be effective they must always be run at build time, and as you add more domain classes this will increase even more and your co-workers will no doubt plot to murder you.

When you are testing POF there is something simpler you can do, something I picked up from Dr. Alberto Forti on a project we worked together a while ago.

You can easily create and use the configurable POF context directly. The ability to do that also opens up some interesting possibilities in using POF with external systems, a topic I will cover on a later post.

Below is an example of a unit test using Configurable POF context: