FTP Client Factory

As the FTP extension for Spring Integration has no official release yet, I have used the latest build which can be found at the Spring Integration Adapters site.

The extension provides a client factory that allows you to connect with a client. The class I have used is the DefaultFTPClientFactory which implements the interface FTPClientFactory.

When you have copied these files to your own project, you can configure a client by using the following code:

    <bean id="defaultClient" class="nl.jteam.importer.ftp.DefaultFTPClientFactory">
        <property value="${ftp.remotedir}" name="remoteWorkingDirectory"></property>
        <property value="${ftp.username}" name="username"></property>
        <property value="${ftp.password}" name="password"></property>
        <property value="${ftp.port}" name="port"></property>
        <property value="${ftp.host}" name="host"></property>
    </bean>

Reading the files

Now that the client is configured, we can read the files from the FTP server. In the method getFilesFromFTPClient() we get a FTPClient by calling the getClient() method on the client factory. The client API provides you with the possibility to retrieve, delete, rename or store files. The API offers a lot more, but I won’t discuss all the methods here. In our case we wanted to retrieve only zip files. As the client does not provide you the functionality to retrieve files from a specific extension, you have to do it yourself by, for example, checking the extension of each file.

Don’t forget to close the connection with the FTP client once you are done with handling files.

Because we want use Spring Integration to send the files to the class that handles the zip files, we create a Message with the list of zip files.

The (partial) code of the FTPInboundAdapter class:

    public FTPInboundAdapter(FTPClientFactory clientFactory, String localTmpDirName) throws IOException {
        Assert.notNull(localTmpDirName, "The directory name to write the files to can not be null");
        this.clientFactory = clientFactory;
        localDirectory = ImportUtils.ensureTempDirExists(localTmpDirName);
    }

    public Message<list><file>&gt; getFilesFromFTPClient() {
        FTPClient client = null;
        try {
            client = clientFactory.getClient();

            List<file> localZipFiles = retrieveRemoteZipFiles(client);

            return MessageBuilder.withPayload(localZipFiles).build();
        } catch (IOException e) {
            throw new MessagingException("Problem occurred while trying to retrieve files.", e);
        } finally {
            closeFtpClient(client);
        }
    }

    private void closeFtpClient(FTPClient client) {
        if (client != null &amp;&amp; client.isConnected()) {
            try {
                client.disconnect();
            } catch (IOException e) {
                logger.warn("Error occurred when disconnection FTP client", e);
            }
        }
    }

The configuration code for the adapter:

    <bean id="ftpInboundAdapter" class="nl.jteam.importer.ftp.FTPInboundAdapter">
        <constructor-arg ref="defaultClient" />
        <constructor-arg value="someLocalDirectory" />
    </bean>

Checking the FTP directory

If you want Spring Integration to check the FTP server on a regular base for new files, you can wire up an inbound channel adapter with a cron expression like this:

    <si:inbound-channel-adapter id="zipInboundChannelAdapter" ref="ftpInboundAdapter" method="getFilesFromFTPClient" channel="zipFilesChannel">
        <si:poller max-messages-per-poll="1">
            <si:cron-trigger expression="0 0/5 * ? * *" />
        </si:poller>
    </si:inbound-channel-adapter>

The channel attribute of the inbound channel adapter specifies the output channel. So in our case this will be the message with the list of zip files that is put onto this channel. This channel can then be used to send the message to wherever you want.

As you could see, it was relatively easy to connect with a FTP server and retrieve the files. I hope this post helped you in setting up your own FTP connection with Spring Integration. All that rests us is waiting for official release of the Spring Integration Adapters.

Maven dependency

If you use Maven, you can simply add the following dependency to use the library.

<dependency>
    <groupid>net.htmlparser.jericho</groupid>
    <artifactid>jericho-html</artifactid>
    <version>3.1</version>
</dependency>

API

I don’t want to explain all classes, but the following classes are basically the starting point of all your parsing.

• Source – Represents a source HTML document. This is always the first step in parsing an HTML document.
• OutputDocument – Represents a modified version of an original Source document or Segment.
• Element – Represents an element  in a specific source document, which encompasses a start tag, an optional end tag and all content  in between.

For a complete overview of all classes you can view the javadoc.

Extract all text

To extract all the text from the HTML markup, all you have to do is the following:

    public String extractAllText(String htmlText){
        Source source = new Source(htmlText);
        return source.getTextExtractor().toString();
    }

You define a new Source object that takes in our case a String as input. But it also accepts for example a InputStream or URL. The Source object contains a method getTextExtractor that allows you to, how surprising, extract the text. The TextExtractor class gives you a few options to configure the extraction. One of the options is that you can exclude text from a specified Element. You can also include an attribute. The value of that attribute will be included in the output.

Manipulating HTML

Manipulating HTML is very easy with Jericho. In the code example below I want to add an id attribute to all H2 elements to create anchor navigation. One again I create a Source document. From this Source document I create an OutputDocument.

The OutputDocument represents a modified version of the original Source document. With the list of all H2 elements retrieved from the Source, we now can ask for all the attributes of a single H2 element. If the attribute id already exists we do nothing, but if it does not we recreate the starttag with a new id attribute and all the other existing attributes from that H2 element.

As you can see in the example, it is relatively easy to manipulate attributes of an element. With the Attributes object you can get a List of Attribute objects that are found in the source document or in a starttag. These attributes are not modifiable. The outputDocument has a convenience method that allows us to replace the specific startTag with our newly created H2 start tag in order to add our id attribute.

    public String addIdAttributeToH2Elements(String html) {
        Source source = new Source(html);
        OutputDocument outputDocument = new OutputDocument(source);
        List<element> h2Elements = source.getAllElements("h2");

        for (Element element : h2Elements) {
            StartTag startTag = element.getStartTag();
            Attributes attributes = startTag.getAttributes();
            Attribute idAttribute = attributes.get("id");

            if (idAttribute == null) {
                String elementValue = element.getTextExtractor().toString();
                String validAnchorId = AnchorUtils.getLowerCasedValidAnchorTitle(elementValue);

                StringBuilder builder = new StringBuilder();
                builder.append("<h2").append(" ").append("id=\"").append(validAnchorId).append("\"");
                for (Attribute attribute : attributes) {
                    builder.append(" ");
                    builder.append(attribute);
                }
                builder.append(">");

                outputDocument.replace(startTag, builder);
            }
        }

        return outputDocument.toString();
    }

Remove Elements

Just like me, you may want to remove a few tags from your HTML. Here is an example that shows you how you can achieve that.

    private static final Set<string> ALLOWED_HTML_TAGS = new HashSet<string>(Arrays.asList(
            HTMLElementName.ABBR,
            HTMLElementName.ACRONYM,
            HTMLElementName.SPAN,
            HTMLElementName.SUB,
            HTMLElementName.SUP)
    );

    private static String removeNotAllowedTags(String htmlFragment) {
        Source source = new Source(htmlFragment);
        OutputDocument outputDocument = new OutputDocument(source);
        List<element> elements = source.getAllElements();

        for (Element element : elements) {
            if (!ALLOWED_HTML_TAGS.contains(element.getName())) {
                outputDocument.remove(element.getStartTag());
                if (!element.getStartTag().isSyntacticalEmptyElementTag()) {
                    outputDocument.remove(element.getEndTag());
                }
            }
        }

        return outputDocument.toString();
    }

In the example above you see that after checking if the tag is allowed, we need to remove the start and endtag. If you would remove the complete element, then you would also remove the text within these tags. The API allows you to check for elements that are empty. This can be handy to remove redundant empty elements or in my case to check if the starttag a self closing tag.

Conclusion

In this post I showed you how I have used Jericho, but Jericho has a lot more interesting features. On their webpage they provide more examples on how to use those features. Jericho provides a nice and clean API and makes the parsing of HTML really easy!

Estimators for recommendations

Let’s start with the main usage of estimators: providing recommendations. Suppose we create a GenericItemBasedRecommender, provide it with a DataModel and one of Taste’s ItemSimilarity implementations.

To fetch a few recommendations we call GenericItemBasedRecommender.mostSimilarItems(long itemID, int howMany), as shown in the snippet below:

  @Override
  public List<RecommendedItem> mostSimilarItems(long itemID, int howMany) throws TasteException {
    return mostSimilarItems(itemID, howMany, null);
  }

  @Override
  public List<RecommendedItem> mostSimilarItems(long itemID, int howMany,
                                                Rescorer<LongPair> rescorer) throws TasteException {
    TopItems.Estimator<Long> estimator = new MostSimilarEstimator(itemID, similarity, rescorer);
    return doMostSimilarItems(new long[] {itemID}, howMany, estimator);
  }

After delegating the method call to a more generic mostSimilarItems method, a MostSimilarEstimator is constructed and passed to the protected method doMostSimilarItems. The whole process of estimating and recommending is implemented via an estimator and algorithm specific logic within a recommender.

Now let’s zoom in on the doMostSimilarItems method. See the snippet below:

  private List<RecommendedItem> doMostSimilarItems(long[] itemIDs,
                                                   int howMany,
                                                   TopItems.Estimator<Long> estimator) throws TasteException {
    DataModel model = getDataModel();
    FastIDSet possibleItemsIDs = new FastIDSet();
    for (long itemID : itemIDs) {
      PreferenceArray prefs = model.getPreferencesForItem(itemID);
      int size = prefs.length();
      for (int i = 0; i < size; i++) {
        long userID = prefs.get(i).getUserID();
        possibleItemsIDs.addAll(model.getItemIDsFromUser(userID));
      }
    }
    possibleItemsIDs.removeAll(itemIDs);
    return TopItems.getTopItems(howMany, possibleItemsIDs.iterator(), null, estimator);
  }

The snippet above describes the core logic for item-based recommendation. This process consists of three steps:

1. Fetch all preferences for the given item(s)
2. For each preference get the corresponding user and fetch all their other preferences
3. From this set of preferences, minus the given item, get the corresponding items and determine the top items based on the given estimator

The TopItems is a helper class for fetching the top ranked items of a set of items for a given estimator.

Now on to the estimator. All estimators implement TopItems.Estimator<T> interface which is really simple. It returns an estimate for a ‘thing’ as a double.

  public interface Estimator<T> {
    double estimate(T thing) throws TasteException;
  }

Now on to the MostSimilarEstimator:

  public static class MostSimilarEstimator implements TopItems.Estimator<Long> {

    private final long toItemID;
    private final ItemSimilarity similarity;
    private final Rescorer<LongPair> rescorer;

    public MostSimilarEstimator(long toItemID, ItemSimilarity similarity, Rescorer<LongPair> rescorer) {
      this.toItemID = toItemID;
      this.similarity = similarity;
      this.rescorer = rescorer;
    }

    @Override
    public double estimate(Long itemID) throws TasteException {
      LongPair pair = new LongPair(toItemID, itemID);
      if ((rescorer != null) && rescorer.isFiltered(pair)) {
        return Double.NaN;
      }
      double originalEstimate = similarity.itemSimilarity(toItemID, itemID);
      return rescorer == null ? originalEstimate : rescorer.rescore(pair, originalEstimate);
    }
  }

This estimator does three things:

1. Use the Rescorer to filter items. Rescorers can be used to create domain specific filtering of items
2. Use the ItemSimilarity to calculate the preference of a user for the given item
3. Optionally boost the similarity value with the Rescorer

This setup allows you to plugin arbitrary ItemSimilarity algorithms in the recommender.

Recommended because…

Another interesting feature of the GenericItemBasedRecommender is the ‘Recommended because’ feature. With this feature you can determine why a certain item was recommended to you, i.e. which of your preferences were largely responsible for giving you this recommendation.

To use this feature call recommendedBecause(long userID, long itemID, int howMany), see snippet below:

  @Override
  public List<RecommendedItem> recommendedBecause(long userID, long itemID, int howMany) throws TasteException {
    if (howMany < 1) {
      throw new IllegalArgumentException("howMany must be at least 1");
    }

    DataModel model = getDataModel();
    TopItems.Estimator<Long> estimator = new RecommendedBecauseEstimator(userID, itemID, similarity);

    PreferenceArray prefs = model.getPreferencesFromUser(userID);
    int size = prefs.length();
    FastIDSet allUserItems = new FastIDSet(size);
    for (int i = 0; i < size; i++) {
      allUserItems.add(prefs.getItemID(i));
    }
    allUserItems.remove(itemID);

    return TopItems.getTopItems(howMany, allUserItems.iterator(), null, estimator);
  }

It takes all items the given user has a preferences for, minus the given item and passes this to TopItems, along the with RecommendedBecauseEstimator, see the code below:

  private class RecommendedBecauseEstimator implements TopItems.Estimator<Long> {

    private final long userID;
    private final long recommendedItemID;
    private final ItemSimilarity similarity;

    private RecommendedBecauseEstimator(long userID, long recommendedItemID, ItemSimilarity similarity) {
      this.userID = userID;
      this.recommendedItemID = recommendedItemID;
      this.similarity = similarity;
    }

    @Override
    public double estimate(Long itemID) throws TasteException {
      Float pref = getDataModel().getPreferenceValue(userID, itemID);
      if (pref == null) {
        return Float.NaN;
      }
      double similarityValue = similarity.itemSimilarity(recommendedItemID, itemID);
      return (1.0 + similarityValue) * pref;
    }
  }

}

This RecommendedBecauseEstimator determines the ranking by multiplying the preference value of the user by the item similarity value of the current item pair. After this process the top ranked items are those items that were most important in causing a recommendation of the given item.

Conclusions

This concludes the overview of some Taste internals and has hopefully given you a clearer picture on how recommendations and estimators work inside Taste. In future posts I will probably expand on this topic, especially within the context the evaluation of recommenders. If you have any questions regarding Taste in general or this topic of estimators feel free to leave a comment.

My colleague Jettro Coenradie explained in a previous post how you can create a feed using Rome and Spring 3, but didn’t elaborated on the Spring 3 part. I will explain how we used Spring 3 to create the feeds.

Context

In this project we need to support a large number of different feeds. As an example I will use the news feed that we need to create for different subjects. The subjects are created in a CMS and therefore people can add subjects whenever they want. This means that we want to support dynamically created feeds.

Controller

The controller is the place where the request mapping takes place.

@Controller
public class SubjectRssController extends AbstractRssController {

    @SessionManagedDaos
    @RequestMapping(value = "/subjects/{subjectName}/news.rss", method = RequestMethod.GET)
    public String generateSubjectNewsFeedItems(ModelMap modelMap, @PathVariable final String subjectName,
                                               HttpServletRequest request, RssSubjectDao subjectDao) throws ServletException {
        SubjectBean subject = findSubject(subjectName, request, subjectDao);

        RssQueryResultMapper mapper = getSubjectQueryResultMapper();
        List<item> items = subjectDao.findNewsForSubject(subject, mapper);

        modelMap.addAttribute("items", items);
        modelMap.addAttribute("subjectName", subject.getTitle());
        return "subjectRssFeed";
}

Annotations

The first thing you might notice when you look at the code above are the two annotations (@SessionManagedDaos and @RequestMapping) on the method. The @SessionManagedDaos is an annotation that is created by us. This annotation manages the session that we use from the CMS connection pool.

The second annotation is the @RequestMapping. This annotation has the value “/subjects/{subjectName}/news.rss” which means that every path that matches this pattern will be mapped to this method. In this pattern uses a new feature of Spring 3, namely the variable in the path. The @PathVariable annotation in front of the parameter subjectName indicates that the variable in the path needs to be bound to the method parameter.

This helps us to support any request with a given subject name and therefore we do not need to change the code if a new subject is created.

Wrong request

With the freedom of the path pattern also comes the problem that a person can try any subject name that he wants. In order to prevent that we execute a query for a subject that does not exist, we execute a query that finds the subject. If the subject is not found, we throw a NoSuchRequestHandlingMethodException. This exception is handled by the DefaultHandlerExceptionResolver. This resolver handles certain standard Spring MVC exceptions by setting a specific response status code. In our case the status code will be a 404.

protected SubjectBean findSubject(String subjectName, HttpServletRequest request, RssSubjectDao subjectDao)
                                  throws NoSuchRequestHandlingMethodException {
    SubjectBean subject = subjectDao.findSubjectBeanByName(subjectName);
    if (subject == null) {
        throw new NoSuchRequestHandlingMethodException(request);
    }

    return subject;
}

Items

The implementation of the controller is not special. We execute a query that finds some news and with the help of a mapper, we map the retrieved beans to RSS items. These items, as well as the subject title, are added to the model.

View

Now that our model is filled, we can create the view. Spring 3 provides us with the AbstractRssFeedView. This view has a few entry points to create the feed.

We created a BaseRssFeedView which extends the AbstractRssFeedView. The BaseRssFeedView overrides three methods that build the feed.

public abstract class BaseRssFeedView extends AbstractRssFeedView {

    private LocalizedMessageResolver messages;

    @Override
    protected void prepareResponse(HttpServletRequest request, HttpServletResponse response) {
        super.prepareResponse(request, response);
       response.setCharacterEncoding(CharacterEncodingConstants.UTF8);
    }

    @Override
    protected void buildFeedMetadata(Map<String, Object> model, Channel feed, HttpServletRequest request) {
        RssMetaData rssMetaData = new RssMetaData();
        configureRssMetaData(rssMetaData, model);

        feed.setTitle(rssMetaData.getFeedTitle());
        feed.setDescription(rssMetaData.getFeedDescription());
        feed.setLink(feedLink);
        feed.setWebMaster(feedWebmaster);
        feed.setLanguage(feedLanguage);
        feed.setCopyright(feedCopyright);

        Image image = resolveImage(rssMetaData.getFeedTitle());
        feed.setImage(image);
        feed.setLastBuildDate(new Date());
        feed.setTtl(TIME_TO_LIVE);

        AtomNSModule module = new AtomNSModuleImpl();
        module.setLink(rssMetaData.getFeedUrl());
        feed.getModules().add(module);
    }

    @Override
    protected List<Item> buildFeedItems(Map<String, Object> stringObjectMap, HttpServletRequest request, HttpServletResponse response) {
        return (List<Item>) stringObjectMap.get("items");
    }

    protected abstract void configureRssMetaData(RssMetaData metaData, Map<String, Object> viewModel);
}

I don’t want to discuss the implementation of the methods, as I think that they are pretty self-explanatory. But I do want to discuss the one abstract method that needs to be implemented by the subclasses. This method allows the subclass to specify the specific metadata for the feed. The code below shows the view that would belong to our SubjectRssController.

@Component("subjectRssFeed")
public class SubjectRssFeedView extends BaseRssFeedView {

    @Override
    protected void configureRssMetaData(RssMetaData metaData, Map<String, Object> viewModel) {
        String subjectName = (String) viewModel.get("subjectName");

        LocalizedMessageResolver messages = getMessageResolver();
        metaData.setFeedUrl(messages.getMessage("subject.news.rss.feed.url", subjectName));
        metaData.setFeedTitle(messages.getMessage("subject.news.rss.feed.title", subjectName));
        metaData.setFeedDescription(messages.getMessage("subject.news.rss.feed.description", subjectName));
    }
}

The class is annotated with @Component(“subjectRssFeed”). The name of the view is subjectRssFeed and needs to match with the view name you return in the controller.

Properties

In this case we assume that every subject has the same message for the specific metadata. So the only thing that differs is the subject name. For example, the description of a feed of the subject cars could be: This feed contains the news items of the subject cars.

So this means that we need to parameterize the messages. In order to do that we have created a class that resolves the message for us from the correct property file. This way we can still use the templates in the property files.

subject.news.rss.feed.description=This feed contains the news items of the subject {0}

Spring 3 also lets you annotate a field or method/constructor parameter that indicates a default value expression for the affected argument. This means you could inject a property like this:

@Value("#{rssProperties.getProperty('subject.news.rss.feed.url')}")
public void setFeedUrl(String feedUrl) {
    this.feedUrl = feedUrl;
}

or

@Value("#{rssProperties.getProperty('subject.news.rss.feed.url')}")
private String feedUrl;

Conclusion

I hope you got an idea of how Spring 3 can help you with creating RSS feeds. In our case it was really useful as we get all the advantages of Spring MVC and added new feeds are a piece of cake!

Besides some changes to existing building blocks, such as the event bus, which is now much more powerful, the 0.5 version also includes some new features.

Read on to find out more.

New features

• Code restructuring. The package structure has been changed to better reflect the different architectural components Axon Framework provides. It should be easier to find the one you’re looking for.
• Command Bus. The command bus is added to Axon. It provides you the ability to explicitly define commands and dispatch them to your command handlers. Furthermore, the command bus provides you the ability to process commands regardless of their type using interceptors. This is useful for, for example, logging, authorization and correlation of incoming commands.
• JPA Event Store. The easiest CQRS configuration is on using full-consistency. That means everything should run within a single transaction. Since transactions over multiple data sources involve a huge performance penalty, Axon provides a JPA Event Store. Its performance is not as good as the FileSystem version, but is does provide transaction support.
• Easy switching between full-consistency and eventual consistency. You can easily choose to process all commands and related events inside a single transaction, or to handle events asynchronously. Choosing for consistency or high-performance is just a matter of configuration. No coding required.
• Per-event listener configuration of asynchronous processing. It is now possible to decide on synchronous vs asynchronous event processing for each event handler individually, just by adding an annotation. If you configure a transaction manager for your event listeners, Axon will process the events in batches and manage the transactions around them.
• Support for rolling snapshots. All event stores will automatically pick up snapshot events. Snapshot events are an important performance booster when aggregates generate a lot of events. Instead of reading all passed events, the event store just needs to read the last snapshot event and the regular events created since the snapshot.
• Transactional Event Processing. Configuring transactions in asynchronous event processing is now a lot easier. 0.5 includes a SpringTransactionManager you can use in combination with Spring’s PlatformTransactionManager.
• Major documentation update. The documentation has been restructured to make it easier to find what you’re looking for.

Maven Central

Where the 0.4 version required configuration of a repository in your project’s pom.xml, the 0.5 version doesn’t. All required artifacts are available in the maven central repository.

Workshop and professional support

We believe that the 0.5 version of Axon Framework is a major step towards production readiness. Therefore, JTeam has decided to provide professional support for the Axon Framework and organize workshops to get you acquainted with the numerous features and choices involved with CQRS.

The first workshop is planned for Friday May 21st in our office in Amsterdam. For more information, visit http://www.jteam.nl/training/workshop/cqrs-axon-framework-training-workshop.html.

Getting started

Want to get started? Visit www.axonframework.org and download the reference guide. That should contain enough information to get you started. If you still have questions, drop me a message.

Online movie store

Our running example for this post is an online DVD shop in which you can view and rent movies. Visitors go to a movie’s page to check out the plot description and will be shown a list of similar movies. These are other movies that were watched by people who also rented that specific movie. Because of space- and time-constraints, this application only provides a view on the dataset and the recommended movies. Other features expected of an online movie rental service, such as registration and payment are left out.

Movielens Dataset

The movies and their ratings originate from the Movielens dataset of the Grouplens research group from the University of Minnesota. There are datasets contains 100.000, 1 million and 10 million ratings. Note that these ratings are explicit, ranging from 1 to 5. This is different from the example in my earlier blog, which featured implicit ratings. Implicit ratings only indicate if a user purchased or liked an item but not how much someone liked it. In this example I used the 100.000 ratings file.

Item-based or user-based algorithms

For this application we use an item-based recommender, which is a good choice performance-wise if the number of items is less than the number of users. This is likely to be the case for an online movie rental store. Item-based recommendation is different from user-based recommendation, which works by identifying a user neighbourhood of similar users and recommending items from the user neighbourhood to other users. User-based recommendations are personalized while with item-based recommendations each user will get the same recommendations for a given item. In our case, all users that visit a specific movie page will see the same recommended movies for that movie. An example of user-based recommendation is Stumbleupon, which provides personalized website recommendations. Stumbleupon requires that you to login first so that it can perform its user-based recommendation based on your profile of likes and dislikes of certain sites.

EuclidianDistanceSimilarity

Now we need to select an item-based algorithm that fits our dataset. The EuclidianDistanceSimilarity is one of Taste’s algorithms that is suitable for explicit ratings and will be used for our demonstration. Before we construct our recommender we introduce the euclidian distance similarity in more detail. The algorithm computes the euclidian distance between each item’s preference vector. The shorter the distance between these vectors, the greater the similarity. For instance, suppose we have users u1, u2 and u3 and item i1. Let’s say these preferences for these users are 2, 4 and 5 respectively. We now have a preference vector [2,4,5] for item i1. The euclidian distance between two of such vectors can now be computed and used as a measure of their similarity. The formula for computing the euclidian distance between two vectors i and j equals the root of the sum of squared differences between coordinates of a pair of vectors. See the formula below:

The EuclidianDistanceSimilarity calculates this similarity for each pair of items and then returns

which results in a value between 0 and 1. The EuclidianDistanceSimilarity can also be weighted. If you pass in the Weighting.WEIGHTED enum to the constructor of EuclidianDistanceSimilarity then the algorithm will weight the values based on the number of users and the number of co-occurring preferences.

Creating a web-based recommender

Below is a list of components we need to create a small web-based recommendation engine, using Taste, Wicket and JPA/Hibernate. I won’t cover all the details of building this webapp, just the main building blocks. You can download the code for this example here and look at the specific details. We need the following components:

• Datamodel
  A FileDataModel that reads movie ids, user ids and ratings from the Movielens dataset directly into memory.
• EuclidianDistanceSimilarity
  Computes item similarities for each pair of items in the datset.
• GenericItemBasedRecommender
  Uses both the datamodel and the similarity algorithm to compute similar items in memory.
• MovieRepository
  JPA repository for retrieving Movie objects
• MovieService
  Uses the recommender and the MovieRepository to retrieve most similar movies for a given movie id.
• Wicket MoviePage, HTML + CSS
  This includes a page for viewing a movie along with similar movies, a few model classes, some HTML and CSS, and a few code tweaks to the original wicket quickstart project.

Note that Taste ships with a preconfigured MovielensRecommender. For the purpose of this article however, I wanted to show you how to build a recommender from the ground up.

Configuring the ResourceDataModel, EuclidianDistanceSimilarity and GenericItemBasedRecommender

Because of license restrictions the movielens data cannot be shipped with this demo, so you need to download it here. Place the u.data file under src/main/resources/grouplens/100K/ratings/ and place the u.item file under src/main/resources/grouplens/100K/data/. Now that you have the ratings data setup you need to feed it into a DataModel class. You can use a FileDataModel for this but for this you need to use an absolute path. Instead I implement a ResourceDataModel which reads ratings files from the classpath. Below is the implementation of the ResourceDataModel, which is a wrapper around a FileDataModel. More on how to wire this below.

package nl.jteam.mahout.gettingstarted.datamodel;

// Imports omitted.

/**
 * DataModel implementation which reads a Spring {@link org.springframework.core.io.Resource} into a
 * {@link org.apache.mahout.cf.taste.impl.model.file.FileDataModel} delegate.
 *
 * @author Frank Scholten
 */
public class ResourceDataModel implements DataModel {
    FileDataModel delegate;

    ResourceDataModel() { // For testing
    }

    /**
     * Reads the preferences from the given {@link org.springframework.core.io.Resource}
     *
     * @param resource with user IDs, items IDs and preferences
     */
    public ResourceDataModel(Resource resource) {
        try {
            this.delegate = new FileDataModel(resource.getFile());
        } catch (IOException e) {
            throw new RuntimeException("Could not read resource " + resource.getDescription(), e);
        }
    }

    @Override
    public LongPrimitiveIterator getUserIDs() throws TasteException {
        return delegate.getUserIDs();
    }

    @Override
    public PreferenceArray getPreferencesFromUser(long userID) throws TasteException {
        return delegate.getPreferencesFromUser(userID);
    }

    @Override
    public FastIDSet getItemIDsFromUser(long userID) throws TasteException {
        return delegate.getItemIDsFromUser(userID);
    }

    @Override
    public LongPrimitiveIterator getItemIDs() throws TasteException {
        return delegate.getItemIDs();
    }

    @Override
    public PreferenceArray getPreferencesForItem(long itemID) throws TasteException {
        return delegate.getPreferencesForItem(itemID);
    }

    @Override
    public Float getPreferenceValue(long userID, long itemID) throws TasteException {
        return delegate.getPreferenceValue(userID, itemID);
    }

    @Override
    public int getNumItems() throws TasteException {
        return delegate.getNumItems();
    }

    @Override
    public int getNumUsers() throws TasteException {
        return delegate.getNumUsers();
    }

    @Override
    public int getNumUsersWithPreferenceFor(long... itemIDs) throws TasteException {
        return delegate.getNumUsersWithPreferenceFor(itemIDs);
    }

    @Override
    public void setPreference(long userID, long itemID, float value) throws TasteException {
        delegate.setPreference(userID, itemID, value);
    }

    @Override
    public void removePreference(long userID, long itemID) throws TasteException {
        delegate.removePreference(userID, itemID);
    }

    @Override
    public boolean hasPreferenceValues() {
        return delegate.hasPreferenceValues();
    }

    @Override
    public void refresh(Collection<Refreshable> alreadyRefreshed) {
        delegate.refresh(alreadyRefreshed);
    }
}

Next you need to initialize the movie database. Run the initialize_movielens_db.sql from the src/main/resources/sql folder. This will create the movielens database and a user with username and password movielens. Additionally, it creates the movie database and loads it with the movie titles from the u.item file.

Now we wire up the GenericItemBasedRecommender and its dependencies, the EuclidianDistanceSimilarity and the ResourceDataModel in the following spring context:

 <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!-- Recommender -->

    <bean id="euclidianDistanceRecommender" class="org.apache.mahout.cf.taste.impl.recommender.GenericItemBasedRecommender">
        <constructor-arg index="0" ref="movielensDataModel100K"/>
        <constructor-arg index="1" ref="euclidianDistanceSimilarity"/>
    </bean>

    <!-- DataModel -->

    <bean id="movielensDataModel100K" class="nl.jteam.mahout.gettingstarted.datamodel.ResourceDataModel">
        <constructor-arg value="classpath:/grouplens/100K/ratings/u.data"/>
    </bean>

    <!-- Similarity -->

    <bean id="euclidianDistanceSimilarity" class="org.apache.mahout.cf.taste.impl.similarity.EuclideanDistanceSimilarity">
        <constructor-arg ref="movielensDataModel100K"/>
        <constructor-arg value="WEIGHTED"/>
    </bean>

</beans>

MovieRepository, MovieService & MoviePage

Our recommender is now ready to determine similar movies for a given movie ID. However, the GenericItemBasedRecommender interface only returns movie IDs of the type long. In order to display the actual movie information to users we need to create a MovieRepository which fetches recommended Movie objects. Additionally, we need a MovieService which coordinates the MovieRepository and the GenericItemBasedRecommender so that we can retrieve recommended Movie objects for a given movie ID. Below is a snippet of a JPA implementation of a MovieRepository.

package nl.jteam.mahout.gettingstarted.repository;

// Imports omitted.

/**
 * Repository for retrieving {@link Movie}s
 *
 * @author Frank Scholten
 */
@Repository
public class JpaMovieRepository implements MovieRepository {

    @PersistenceContext
    private EntityManager entityManager;

    /** {@inheritDoc} */
    @Override
    public Movie getMovieById(long id) {
        return entityManager.find(Movie.class, id);
    }

    /** {@inheritDoc} */
    @Override
    @SuppressWarnings("unchecked")
    public List<Movie> getMoviesById(List<Long> movieIds) {
        return (List<Movie>) entityManager.createQuery("SELECT m FROM movie m WHERE m.id IN (:movieIds)")
                .setParameter("movieIds", movieIds)
                .getResultList();
    }
}

Below is a code snippet of a default implementation of the MovieService.

package nl.jteam.mahout.gettingstarted.service;

// Imports omitted.

/**
 * Service for retrieving and recommending {@link Movie}s.
 *
 * @author Frank Scholten
 */
@Transactional
@Service
public class DefaultMovieService implements MovieService {

    @Autowired
    private MovieRepository movieRepository;

    @Autowired
    private ItemBasedRecommender movieRecommender;

    public Movie getMovieById(long id) {
        return movieRepository.getMovieById(id);
    }

    @SuppressWarnings("unchecked")
    public List<Movie> moreLikeThis(long movieId) {
        try {
            List<RecommendedItem> recommendedItems = movieRecommender.mostSimilarItems(movieId, 5);

            List<Long> ids = new ArrayList();
            for (RecommendedItem r : recommendedItems) {
                ids.add(r.getItemID());
            }

            return movieRepository.getMoviesById(ids);

        } catch (TasteException e) {
            return (List<Movie>) Collections.EMPTY_LIST;
        }
    }
}

Finally, below is the snippet of the Wicket MoviePage which displays the current movie and similar movies fetched through specially created Wickets models.

package nl.jteam.mahout.gettingstarted.web.page;

// Imports omitted.

/**
 * Page for showing a single {@link Movie} from the Movielens dataset along
 * with recommended movies i.e. 'more like this'.
 *
 * @author Frank Scholten
 */
public class MoviePage extends WebPage {

    private static final String MOVIE_ID = "0";

    public MoviePage(PageParameters pageParameters) {
        final long movieId = pageParameters.getLong(MOVIE_ID, 1);

        MovieModel model = new MovieModel(movieId);
        add(new Label("title", model.getObject().getTitle()));

        PropertyListView<Movie> recommendedMovies = new PropertyListView<Movie>("moreLikeThis", new RecommendedMoviesModel(movieId)) {
            @Override
            protected void populateItem(ListItem listItem) {
                Movie movie = (Movie) listItem.getModelObject();
                PageParameters pageParameters = new PageParameters();
                pageParameters.put(MOVIE_ID, movie.getId());

                BookmarkablePageLink<MoviePage> movieLink = new BookmarkablePageLink<MoviePage>("link", MoviePage.class, pageParameters);
                listItem.add(movieLink);
                Label movieTitle = new Label("title");
                movieTitle.setRenderBodyOnly(true);
                movieLink.add(movieTitle);
            }
        };
        add(recommendedMovies);
    }
}

Running the web application

First you need to download the Movielens dataset and add the ratings file on the classpath under grouplens/100K/ratings. See the spring context above. Since this example is based on the Wicket quickstart project you can start the application via Jetty through the Start class and run it in your favourite IDE. Go to http://localhost:9090/ and you can browse through movies via recommendations. Alternatively you can build the WAR and drop it into tomcat.

Performance tweaks

• If you like to experiment with the larger datasets you need to add -Xmx512m as a VM parameter if you want to run this application with the 10 million ratings dataset. Also, Tastes FileDataModel uses commas and tabs as delimiter. You may need to run the Movielens files through sed before feeding them into a FileDataModel
• The FileDataModel reads everything in memory before computation. This is way faster than using a MySQLJDBCDataModel, since this requires around O(n²) database queries to compute similarities between all pairs. Reading all data in memory is not always feasible however. An alternative is to precompute the similarities for the item pairs and store the results in the database and read them via the MySQLJDBCItemSimilarity.
• You can also sample the dataset and/or remove noise elements to speed things up a little more

These performance aspects are an interesting topic for a later blogpost. If any of you reading this has experience with these type of issues please post a comment and we’ll discuss them.

Conclusions

This concludes the getting started post on Mahout / Taste. What we didn’t cover was how to update the recommender and how to customize your recommender with boosting of items. These are all subjects for future blog posts.

References

• Getting started demo – source code

You can download the source code of this example here.

• Grouplens datasets

The Grouplens research group of the University of Minnesota have made a few datasets publicly available for research purposes.

• Mahout in Action EAP

This is a great resource on Mahout and explains a lot about performance issues and details how the algorithms stack up against eachother. Also it provides a lot of examples and case studies on how to use Mahout in practice.

From a user’s perspective

I want to highlight a few of the new features I think are most interesting which come with the new iPhone OS 4.0.

Multitasking

I was always one of the people that believed that the iPhone doesn’t need multitasking at all. I was perfectly happy with the fact that an app can manually save it’s state on exit, and restore that same state when you start the app again, like nothing happened. A while ago I was using a phone that did support multitasking. It was the HTC Magic which was running on Google’s Android platform. I soon noticed the downsides of the ability to multitask. The battery runs out very quickly (even quicker than an iPhone’s battery) and the phone tends to get really sluggish when you have just a few apps running in the background.

The downsides of multitasking I just described were one of the reasons why Apple didn’t support multitasking in the first place. They wanted to make a phone that provided a great user experience which runs smoothly in every circumstance. And also they didn’t want the battery to die in just half a day. But now with OS 4.0, Apple includes multitasking. It’s implemented in a way which preserves battery life and performance! Sounds very promising, doesn’t it? Let’s see if it is too good to be true.

I ran OS 4.0 on the iPhone simulator which comes with the iPhone SDK to play around with this new functionality. Because, like I said, I can’t use the multitask feature on my iPhone 3G. Now I can only demonstrate what the user interface looks like on the simulator. I deployed a little iPhone app which I created a while ago for an existing website. This website was created by a friend of mine and is kind of like a forum with just one thread, used by a group of people that know each other in real life. The screenshots below show how you can easily switch between that app and a Safari web page, as an example. To switch you can double click the home button and the icons of all currently running apps slide into the screen. Quickly switch to one of the running apps by tapping it’s icon.

The way they implemented this is once an app is put in the background, the state of that app freezes. Once you switch back to that app, the app instantly continues to run from the same state you left it in. This way the app doesn’t keep running in the background which would negatively affect battery consumption and performance. Of course, some apps still want to be able to do stuff while in the background. Apple provides those apps with a bunch of services they can utilize to be able to do their thing, while preserving battery life. Here are a few examples:

• Playing music in the background. For example the Pandora Radio app (which I believe isn’t available outside of America unfortunately) is now able to continue to play music even when the app is not running. It’s just like what the iPod app already could do, so now that same technique is available for all music playing apps.
• Voice over IP in the background. For example the Skype app can now receive calls while it’s in the background! This service is kind of similar to what the Phone app already could do. This is awesome because this was the main drawback of using Skype on iPhone. You really needed to have the app running to be able to receive phone calls. Once you quit the app, you could no longer receive phone calls. This service makes it possible so that’s exciting news!
• Receive GPS data while in background. This still consumes some of the battery’s life but it allows for example TomTom, to continue giving the user directions, even when the user is switching to other apps. But since you always have the iPhone connected to a power adapter when using TomTom, the extra battery consumption of this service is no problem at all.

There are a few more services available like this, but the ones I described above are the ones I think are most interesting.

Setting backgrounds

Yes, Apple finally gave in, the support for displaying a background image behind your application icons. So now people that didn’t jailbreak their phones can enjoy a nice little picture in the background too This feature is also only supported on iPhone 3GS. Why it isn’t on 3G? I have no clue at all.

Organizing apps with folders

Another new feature is the ability to create folders to organize your apps! Each folder can contain up to 12 apps. This is a feature that IS supported on my iPhone 3G so I made some screenshots without the need for the simulator this time.

Before OS 4.0 you could “only” install  a total of 180 apps on your phone, so now in theory if you would replace every icon with a folder and stuff it with apps, you could install a total of 2160 apps! Which is totally ridiculous if you would do that, but it IS possible now

Game Center

Apple included a preview of Game Center in the OS 4.0 beta, which will not be in the first release of OS 4.0. It’s supposed to be available later this year for everyone to use. As far as I understand, Game Center is a way for all iPhone games to easily find other users to play with and to keep online score boards and stuff like that. You can even challenge other players who are not currently playing the game. They get a push notification with the challenge, and if the user accepts, the game starts. To me this whole thing sounds just like the iPhone’s version of the Playstation Network and XBOX Live, which I think is awesome

From a developer’s perspective

OS 4.0 also includes a lot of new API’s for developers to work with to make their apps even more awesome.

Local notifications

Since OS 3.0 we already have push notifications, which let a server send messages to your phone when certain events occur. Now, we also get local notifications. The main difference is that these doesn’t require a server. An app can now simply schedule a local notification to be displayed at a certain time. I think this is a really cool feature since you don’t always need a server to display notifications.

Calendar access

By using the new EventKit framework, apps can now access the user’s calendar. So apps can now get existing events from your calendar and add new events to it. This was always something I was missing, but now it’s here!

Quick Look

With the Quick Look framework, apps are now able to show a preview of certain files. For examples attachments to an e-mail, or files in your DropBox. The following file types are supported: iWork documents, Microsoft Office documents, RTF, PDF, images, text and CSV files.

Sending SMS messages

Previously you could send SMS messages only by leaving the app completely and go into the Messages app (this could be directly triggered from within the app). Now SMS messages can be sent without leaving the app by using the Message UI framework. You stay in the same app while typing the text message.

Block object

This actually isn’t a new framework, it’s a new language feature in Objective C! The developer documentation describes it best: “A block object is a mechanism for creating an ad hoc function body, something which in other languages is sometimes called a closure or lambda”. This can for example be used as a replacement for delegates or it can be used as callback functions.

An example to sort a NSString array using a block object:

NSArray *stringsArray = [NSArray arrayWithObjects:@"string 1", @"String 21", @"string 12", @"String 11", @"String 02", nil];
static NSStringCompareOptions comparisonOptions = NSCaseInsensitiveSearch | NSNumericSearch | NSWidthInsensitiveSearch | NSForcedOrderingSearch;
NSLocale *currentLocale = [NSLocale currentLocale];

NSComparator finderSortBlock = ^(id string1, id string2) {
    NSRange string1Range = NSMakeRange(0, [string1 length]);
    return [string1 compare:string2 options:comparisonOptions range:string1Range locale:currentLocale];
};
NSArray *finderSortArray = [stringsArray sortedArrayUsingComparator:finderSortBlock]);
NSLog(@"finderSortArray: %@", finderSortArray);

The NSComparator finderSortBlock is the block object in this example. It describes a function which can compare two strings with each other.

That’s all folks!

That’s all I wanted to share for now. I am very excited about all this new development. Let me know what you think by leaving a comment in the comment section below.

Het is nu ongeveer anderhalf jaar geleden dat ik ben begonnen als Software Architect bij het project Overheid Nieuws Stijl. Het is een ambitieus project om 16 websites samen te laten komen in één rijksbrede website www.rijksoverheid.nl. Met de eerste live gang zijn er 7 websites over: postbus 51, regering.nl, minaz.nl, minez.nl, minocw.nl, minvws.nl en jeugengezin.nl. Uiteraard moest de meeste content van de websites ook op de nieuwe website terug te vinden zijn.

Vanuit de overheid is er voor gekozen om zoveel mogelijk Open Source software te gaan gebruiken. Ook voor het project Overheid Nieuwe Stijl (ONS) is er gekozen voor meerdere Open Source oplossingen. Naast het gebruik van Open Source software worden zaken als Open Standaarden en ook Open Data belangrijk gevonden. Vooral Open Data wordt door iedereen als zeer positief ervaren. De licentie voor alle content is Creative Commons.

Mocht je meer informatie willen hebben over dit enorme project, dan kun je deze hier vinden.

In deze blogpost wil ik me vooral concentreren op de betrokkenheid van JTeam bij dit ambitieuze project voor de rijksoverheid.

Proof-Of-Concept (Poc)

Voor mij startte het project ergens in September 2009. Samen met twee mensen vanuit de overheid een iemand van Hippo zouden wij het Hippo cms op de pijnbank leggen tijdens een Proof-Of-Concept (PoC). Voor deze PoC was er al veel onderzoek gedaan naar content management systemen, daar is Hippo uitgekomen als de beste keuze voor de eisen van de nieuwe rijksoverheid website. Al tijdens de PoC bleek dat Hippo graag wilde helpen om een succes te maken van dit project. Ze hebben vanuit Hippo tijdens de PoC en uiteraard ook tijdens de rest van het project veel features opgeleverd die wij als rijksoverheid nodig hadden. Tijdens de PoC heb ik geholpen de oplossing te realiseren. Waar nodig ben ik kritisch richting Hippo geweest. Heb soms zelfs om verbeteringen gevraagd. Dit heb ik en uiteraard ook mijn collega’s tijdens het project vaker gedaan. Zo hebben we bugs gerapporteerd, soms ook wat patches gemaakt om de uiteindelijke oplossingen beter te maken.

Een klein onderdeel van de PoC was het importeren van content en beschikbaar stellen van data. Door gebruik te maken van spring-ws en spring integration en een eigen gemaakte hippo connector bleek dit heel goed mogelijk.

Met hard werken is de PoC uiteindelijk geslaagd en konden de plannen voor het echte project worden gemaakt.

De aanbesteding

In afwachting van de aanbesteding is het project team verder gegaan met het maken van plannen en voorbereiden van het project. Nadat JTeam bij een van de 6 partijen hoorde is het team uitgebreid en inmiddels zitten Roberto, Rob en Tom ook in het team. Daarmee hebben we als JTeam een behoorlijke bijdrage geleverd aan de uiteindelijke oplossing.

Open Source

Zoals eerder al gezegd is er de voorkeur om zoveel mogelijk met Open Source te werken. Het is bekend dat we gebruik maken van Hippo als content management systeem. Hippo werkt mee in het project, de samenwerking met Hippo heb ik als zeer prettig ervaren. We houden elkaar scherp en vullen elkaar duidelijk aan qua kennis.

Naast hippo zijn er ook nog andere Open Source technologiën waar we gebruik van maken. De belangrijkste die ik nog wel wil noemen zijn twee spring framework projecten. We hebben spring webservices gebruikt voor het accepteren van nieuwe content en content updates. Het afhandelen van nieuwe content is een complex stuk dat we uiteindelijk oplossen met spring integration. Deze technologiën zijn onmisbaar in het hele content migratie stuk. Daarnaast maken we ook gebruik van de RSS mogelijkheden die het spring framework out-of-the-box ondersteund.

Data migratie

De data van de websites wordt middels een commerciële tool geïmporteerd. Kapow van kapowtech is een gespecialiseerde tool voor het lezen van bestaande websites. Zij spreken vervolgens een door ons gerealiseerde webservice aan om de content in Hippo te krijgen. Dit heeft zich tot nu toe bewezen als zeer efficiënt.

Web richtlijnen

Vanuit het ministerie van Volksgezondheid, Welzijn en Sport zijn een aantal mensen binnen gekomen die veel doen met de web richtlijnen. Twee van hen zijn nauw betrokken bij bij het tot stand komen en dus ook naleven van de web richtlijnen. Ik moet eerlijk toegeven dat ik me hier nooit zoveel mee bezig heb gehouden. Natuurlijk kende ik de basis wel, maar ik heb al best wel aardig wat geleerd van deze mannen. Ik heb begrepen dat we nu al nagenoeg voldoen aan de nieuwe versie van de web richtlijnen die momenteel wordt vastgelegd.

Hosting en infrastructuur

Zonder een goede infrastructuur is het onmogelijk om een goede site op te leveren. We maken al lang gebruik van een specialist op het gebied van infrastructuur. Hij komt van het bedrijf prolocation. Samen met een collega heeft hij mij echt weten te verbazen. De kennis die deze twee gasten hebben is enorm. Heb veel van hen opgestoken.

De hosting ga ik het niet te veel over hebben. Hier is het nodige om te doen. Weet wel dat we er erg veel energie in hebben moeten stoppen om de infrastructuur zo te krijgen dat het voor ons project werkt zoals het zou moeten werken.

Open Data

Ik heb al eerder gezegd dat Open Data erg belangrijk is voor het project. In de huidige versie hebben we nog niet zo heel veel gedaan aan het beschikbaar stellen van data. Het gebruik van Creative Commons is wellicht het belangrijkste. Wel is er gestart met het beschikbaar stellen van rss feeds. Uiteraard zullen hier in de toekomst meer features worden opgeleverd.

Persoonlijk moet ik er nog wel eens aan wennen. Ik was dan ook nogal skeptisch over het initiatief van een andere partij om toch meer feeds aan te bieden. Vanuit het project was men het hier echter helemaal niet mee eens en vond dit juist een mooi initiatief.

http://rijksoverheid.jijendeoverheid.nl/

De mensen

Als laatste wil ik toch nog even in gaan op de mensen die in het project zitten. Ik vind echt dat we een enorm goed team hebben weten te creëren met mensen die overal vandaan kwamen. Ambtenaren en externen gaan samen volledig voor een resultaat dat er mag zijn.

Het resultaat

Super snelle website en een gaaf project.

Wat we binnen dit project voor elkaar krijgen kan alleen met een team, en dat is precies wat we met zijn allen zijn geworden. Ik ben enorm trots op dat ik mee heb mogen werken met dit project en ik denk dat ik nog wel even mee blij werken om ook de volgende fases mee te kunnen maken.

Introduction

To finish the last part for my Bachelor of ICT, I recently started my internship at JTeam. JTeam has been doing a lot of projects using either a CMS (e.g. Magnolia and Hippo) or requiring a search solution, typically using Apache Lucene and Apache Solr. My assignment is to investigate the problem of making the information from different CMS search-able using Solr and hopefully come up with a good solution.

In order for the content in a CMS to be available for searching, it needs to be indexed by Solr. Problem is that many of the Java-based CMS solutions do not provide an easy way to get their content to Solr. We need to get the information out of the CMS and feed it to Solr. In this blog post I will discuss the synchronization problem at a more generic level and look at some possible solutions.

Synchronization problem

Before diving into CMS solutions specifically, let’s look at a more classic problem: the data replication problem. When we get information from a specific source and replicate it into another system, the data may get out of date when changes in the original source are not immediately propagated. This is a common problem, that has been studied extensively. Junghoo Cho and Hector Garcia-Molina studied this problem with the focus on Web Crawlers. They started their study by looking how to measure the problem and defined the freshness and age of a database.

Intuitively, we consider a database ”fresher” when the database has more
up-to-date elements. For instance, when database A has 10 up-to-date elements out of 20 elements, and when database B has 15 up-to-date elements, we consider B to be fresher than A. Also, we have a notion of ”age”: Even if all elements are obsolete, we consider database A ”more current” than B, if A was synchronized 1 day ago, and B was synchronized 1 year ago.

With this notion of Freshness and Age we can look at the potential quality of the solutions to this problem.

Possible ways

Basically, there are three ways of replicating the content in a CMS to a search solution like Apache Solr:

1. Look at the CMS’s data to see what has changed
2. Listen to the CMS and hear what has changed
3. Let the CMS update Solr

The first option, looking at the CMS’s data to see what has changed, can be done by creating a crawler. The crawler will iterate over all the content in the CMS and compare the last known version with the current version of the content in the CMS. Such a crawler is typically implemented by repeating the following steps:

• Put all the known elements in a Queue
• While the Queue is not Empty
• Take an element from the Queue
• Check if the element has changed

This is fairly straightforward to implement. Luckily, the same study shows there is little to gain in trying to optimize the order in which we look at the elements, or the amount of resources we use for specific elements (e.g. the rate at which specific elements are checked). However, in order to keep your Solr data fresh and young, a crawler will need more and more resources when the CMS grows.

The second option, listening for changes in the CMS, would be better. In order to listen for changes the CMS should push the changes to external systems. A push, or observation, mechanism can be implemented in one of two ways: asynchronous or journaled. The simplest form, asynchronous observation, will send a notification on every change that occurs in the CMS. Getting notified of the change and being able to incorporate that change will give us a fresh and young Solr index. Although asynchronous observation looks perfect at first sight, it has two flaws: it doesn’t guarantee that all changes are actually sent and does not guarantee the right ordering of the changes. Missing a change, or getting two updates in the wrong order, leaves an element out-of-date, and the Solr index not fresh.

Journal based observation solves the problems of asynchronous observation, as it can guarantee complete information and ordering. Instead of being informed on every change, we will get a list, a journal, of changes since the last time we checked. Using journal based observation, the age is only influenced by the time it takes to process the changes and the freshness by the amount of changes in each journal.

The third option, let the CMS update Solr, is obviously the most ideal. Instead of crawling or observing the CMS and update Solr when something changes, the CMS could directly propagate all changes to Solr. The age and freshness of Solr would be perfect, as changes to data can be made available at the same time to both the CMS and Solr. Having a CMS that is built using an event-driven, highly decoupled architecture, for instance using a CQRS framework, like Axon Framework, would make extending the CMS with the capability for updating Solr very easy. However, currently no Java based CMS provides this capability out of the box.

Conclusion

Trying to solving the problem of making the information from different CMS solutions search-able using Solr gave some interesting insights in the classic data replication problem. Using Age and Freshness as a guideline, letting the CMS update Solr looks like the best solution, but is currently not usable when looking at different CMS solutions. Both asynchronous observation and crawling have up and downsides, but seem to be the most generic solution. Journaled observation, when looking at the Age and Freshness and different CMS systems, looks the most promising.

In the next blog posts as part of this series, I will go into more detail of actually implementing this integration for a number of widely used CMS solutions like Magnolia and Hippo. Stay tuned…

References

Effective Page Refresh Policies For Web Crawlers
Axon framework

The thing is that both Ms Fast and Endeca did not implement their language analysis themselves. They use an existing, commercial solution called the Rosetta Linguistics Platform (RLP) provided by Basis Technologies under the hood to provide their sophisticated language analysis capabilities. This is a good thing, as RLP also provides integration components for Apache Solr. This allows anyone using Solr to easily plug in RLP advanced language capabilities into their solution.

What is RLP?
The Rosette Linguistics Platform (RLP) is a commercial solution that allows you to perform linguistic analysis of text in many languages (English as well as dozens of major European, Asian, and Middle Eastern languages). Besides that RLP also supports advanced entity extraction capabilities, base noun detection, sentence boundary detection and even part of speech tagging. However, as RLP is a commercial product it comes with a price tag.
However, we feel that RLP has a lot of potential when you are in need of sophisticated language capabilities. That’s why we want to show you how to integrate RLP into your solution based on Apache Solr.

Installing the RLP platform for Solr
RLP and Solr are two separate systems. Integration between Solr and RLP is quite straight forward. Basis provides extensive documentation on how to setup the RLP on your machine. With the RLP configuration you can customize the actual language analysis. This is not done directly via Solr. Installing and configuring RLP is explained in RLP documentation that is included with RLP bundle.

Configuring Solr to use RLP for analyzing your documents is also relatively easy to setup. In order to use RLP’s language analysis in Solr you have to configure the RLPTokenizerFactory as tokenizer for the specific fields you want to use it on. As RLP is not written in Java, JNI is used to integrate a Java application with RLP. In order for Solr to integrate with RLP you must set certain environment variables that point to the RLP installation. Basis has quite extensive documentation for the Solr integration with RLP as well, which is included in the RLP Solr bundle.

RLP’s stemming capabilities
Terms like verbs, nouns and adjectives appear in different forms. When searching the different forms of a term, it can result in a miss. For example if you search for the term customize, but the token in the index is customization, your are unlikely to find results with this term. The solution to this is stemming. Stemming reduces a term to a base form. There are many ways to do this but the most common way is to remove the terms’ suffix according to some basic rules. In our case customize and customization would be stemmed to customiz. Stemming is usually applied during indexing and during searching (stemming the search query). So whether we search for customization or customize does not matter, for both terms we’ll get the same result. Another example is go and going, both forms are stemmed to go.

In all languages there are terms that are irregular. For example the verb to buy. The present tense is buy and the the past tense is bought. Using a suffix based stemmer, will not create a common base form for the tokens. The create a proper base form the stemmer needs to have some basic knowledge of a language. Stemmers who do this usually use lemmatisation. Using this algorithm buy and bought would both be stemmed to buy. Lemmatisation based stemmers will stem more terms to a common base form and therefore increase findability of documents in your index.

RLP’s stemming capabilities fall into the last category of stemming algorithms and as you can see in the table below the stemming is quite powerful.

Advantages of using RLP
Why would you use RLP in conjunction with Solr? RLP provides you with really powerful language analysis. Solr is a extensible open-source search engine. By integrating RLP with Solr, RLP will compliment the language analysis provided by Solr and increase the quality of your search results. As seen in the previous table, the better stemming capabilities increases the likelihood for relevant documents to be found.

If you consider using RLP for your project or solution, please feel free to contact us, so we can help you both to make a decision as well as help you implement it.