Contents

• Go live immediately
• Focus on as little use cases as possible
• Be viral
• Forget pixel perfect, improve as you go
• Choose the right technology

Go live immediately

You’ll need to go live as soon as possible. Each and every one of our clients with a social website thought they knew what their clients wanted. They were all wrong (initially). Of course not totally-of-the-track-wrong, but still wrong enough to have wasted valuable time and money. If you go live immediately, you and your users immediately gain insight into how the site works and what you need to do first to improve it.

Lesson learned: You will only know which use cases are really important if your site is being used by real users and they start to provide you with feedback.

Focus on as little use cases as possible (a variation on the previous)

Again, in all of the social websites JTeam has build, our client had a long list of use cases that needed implementing. This obviously translated into an even longer list of features. However, not all of those features were used or even understood by the users. In the majority of sites, we even had to take out features again later.

In combination with the previous lesson learned, the idea of Perpetual Beta can be used for developing your site. This basically means that you release your site early and do quick releases to enhance and extend the site gradually over time.

Lesson learned: You will only know how your site will be used when it is actually being used. At this time it will become apparent what features are useful, which are not. Adding features upfront is not only wasteful, it is counter productive.

Be viral

A common misconception is that a social website needs users. Wrong! You need lots and lots of users, not a thousand, but multiple thousands at least. But how do you get them? The best way is to go viral; you need your users to attract even more users. There are several ways to achieve this. One old fashioned way is some kind of incentive program (expressed as status or plain money). Another status raiser for your site might be a cool mobile application (e.g. iPhone/iPad, Android) that extends the site. Another, probably much much more interesting option, is to build widgets for other social websites like Hyves, Facebook and LinkedIn. I would even go so far as to build the site to serve this user group first. Also using mechanisms to make it easy for users of those other social websites to start using your site will help in adoption. An example of this is using Facebook Connect to allow users to log into your site by using their Facebook credentials.

Lesson learned: Often, features alone are not enough, create attention by hooking into other (big) social sites.

Forget pixel perfect, improve as you go

The first users are not so critical. They like it if they see a continuous stream of small improvements. It makes them feel at home. This is the attitude you like to foster. Forget IE 6 as well, your target group is not likely to use this ancient browser anyway.
Often, our clients focus on pixels rather than functionality and don’t want to release until each and every page is pixel perfect. This leads to a lot of time (and money) being spent on layout rather than functionality.

Lesson learned: Don’t stop a release because there is a small layout problem. Fix it in the next release.

Choose the right technology

You will need technology that is easy to use now, but that won’t bog you down as your site grows. Large site require dis-proportionally more money then small sites. In particular we noticed that relational databases and transaction script architectures are in the way for the type of changes that are required for growing a site. We learned that scalable technologies like Solr (search) and Axon (event based architecture) are better suited for this.

Besides choosing technologies that scale well and are future proof, obviously you also want to make the right choice for technologies that actually do what you want it to do… Especially when you rely fully on a technology partner, make sure that you validate their choices. We have seen numerous cases where our clients came to us with an already developed site that simply used the wrong technologies. This even goes as far as a frontend technology choice for a website that wasn’t indexable by Google, while being findable in Google was of the utmost importance from a business perspective. Each technology has a home ground where its suitable and making the wrong choice there can be disaster.

Lesson learned: Making the right technology choices is essential to ensure scalability and maintainability, but also essential to getting a system that serves your business. Make sure you have a solid technology partner with a track record of choosing the right technologies and ensure you have (or hire) the expertise to validate their choices.

Conclusion

The above lessons learned will help you be more successful in developing your social website if you keep them in mind. Obviously, JTeam can help in advising you, even before you start the implementation. If you are thinking about a social business concept, feel free to contact us.

In this blogpost I will explain the changes I made to the connection pool. After that I’ll show what to do to make this changed connection pool available to a site created with the HST and I’ll show a groovy script that reads the data from the remote servers using the JMX connection.

I started writing about JMX, groovy and spring on my personal blog. I wrote about exposing jmx through spring. Adding groovy spring to the spring context and a bit more about JMX. Check these two posts:

http://www.gridshore.nl/2010/06/02/using-jmx-within-a-spring-application/

http://www.gridshore.nl/2010/06/20/exposing-jmx-through-jmxmp-and-reading-the-jmx-data-with-groovy/

In this blog post I’ll give more details to use this in a Hippo environment.

Changes to the connection pool

Hippo provides the hst-session-pool and the hippo-ecm-repository-connector. The names a re descriptive enough to understand the goals for the two components. In this blog post I am particularly interested in the hst-session-pool. If you are creating an HST website, the spring configuration is configured in the SpringComponentManager-jcr.xml. This is not handled the same as a normal spring web application. But Hippo provides a mechanism to create your own instance of this file and overload the one that is provided by Hippo. More on this later.

For now let us have a look at the class we have created to replace the org.hippoecm.hst.core.jcr.pool.BasicPoolingRepository as provided by Hippo. I am not going over all the methods. I will give you an idea of the changes I made. First an image with the components that are involved.

The blue block is a changed Hippo class, the yellow ones are extensions to support monitoring. PoolingStatistics is used to track statistics like the amount of sessions that are obtained and the amount of sessions that are returned. The BasicPoolingRepositoryWihtLogging uses this statistics object to store this data and exposes it so other can read the data. The PoolingRepositoryMonitor reads the required data from the other two components and exposes this using JMX.

To get an idea of the type of changes I made, have a look at the following code block. The first method is the one from hippo, the second the one that I changed. Look at the use of the statistics object and the logging.

    public Session login() throws LoginException, RepositoryException {
        Session session = null;

        try {
            session = (Session) this.sessionPool.borrowObject();
        } catch (NoSuchElementException e) {
            throw new NoAvailableSessionException("No session is available now. Probably the session pool was exhasuted.");
        } catch (Exception e) {
            throw new LoginException("Failed to borrow session from the pool.", e);
        }

        return session;
    }

    public Session login() throws LoginException, RepositoryException {
        log.debug("Request session by doing a login.");
        try {
            Session session = (Session) this.sessionPool.borrowObject();
            log.debug("Return a session with object id {} from the pool after a login", session.hashCode());
            statistics.sessionObtained();
            return session;
        } catch (NoSuchElementException e) {
            log.debug("Problem obtaining session from the pool", e);
            throw new NoAvailableSessionException("No session is available now. Probably the session pool was exhausted.");
        } catch (Exception e) {
            throw new LoginException("Failed to borrow session from the pool.", e);
        }
    }

Now we have the altered connection pool class, we have the class that can be exposed through JMX using Spring. Next step is to tell the HST to use this class. Look for the file SpringComponentManager-jcr.xml, make a copy and put this copy in the folder (taken that you use maven to build your project) src/main/resources/META-INF/hst-assembly/overrides. Look for all usages of org.hippoecm.hst.core.jcr.pool.BasicPoolingRepository and replace these with the name of your own implementation. At the time of writing there are 5 different repositories in use by the hst. Next up is to initialize the PoolingReposityMonitor beans. We create a bean for each repository bean, so we have 5 monitors. The final part in the spring configuration is to expose the jmx beans using the mBeanServer. Spring provides some beans for that as well. The following code block show the most important beans.

    <context:mbean-server/>
    <context:mbean-export registration="replaceExisting" server="mbeanServer"/>
    <bean id="jmxServerConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean">
        <property name="threaded" value="true"/>
        <property name="daemon" value="true"/>
        <property name="server" ref="mbeanServer"/>
        <property name="serviceUrl" value="${jmx.serviceurl}"/>
    </bean>
    <bean id="defaultJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
        <constructor-arg ref="defaultPoolingRepository" index="0"/>
        <constructor-arg value="readonly" index="1"/>
    </bean>
    <bean id="previewJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
        <constructor-arg ref="previewPoolingController" index="0"/>
        <constructor-arg value="preview" index="1"/>
    </bean>
    <bean id="hstConfigJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
        <constructor-arg ref="hstConfigReaderPoolingRepository" index="0"/>
        <constructor-arg value="hstConfig" index="1"/>
    </bean>
    <bean id="writeableJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
        <constructor-arg ref="writeablePoolingRepository" index="0"/>
        <constructor-arg value="writeable" index="1"/>
    </bean>
    <bean id="binariesJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
        <constructor-arg ref="binaryPoolingRepository" index="0"/>
        <constructor-arg value="binaries" index="1"/>
    </bean>

It is now possible to use a jmx client like jconsole to have a look at the beans. But that is not what I want. I want to connect remote using groovy and I want a nice page that show the basic information. The hst has the possibility to using spring mvc within the hst. I am not going into details on how to do this. Woonsan has written an excellent blog post that explains how to do this.

http://blogs.onehippo.org/woonsan/2009/06/spring_web_mvc_framework_suppo_1.html

I have configured a controller that has a dependency on a groovy bean. The groovy bean created a few instanced of a class MonitorGroup that just contains a title and a map of parameters. The first code block shows the spring configuration. The second code block shows the groovy part of reading the jmx beans for the hippo session pools.

    <bean class="nl.rijksoverheid.monitoring.JmxController">
        <constructor-arg ref="jmxMonitor"/>
    </bean>

    <lang:groovy script-source="classpath:groovy/JmxMonitor.groovy" id="jmxMonitor">
        <lang:property name="mbeanServer" ref="mbeanServer"/>
    </lang:groovy>

    private MonitorGroup createHippoConnectionPoolMonitorGroup() {
        def hippoPoolingQuery = new ObjectName('package.name.of.monitor:*')

        MonitorGroup monitorGroup = new MonitorGroup() {

            List<String> getColumns() {
                return ["Name", "Type","Active", "Idle", "Obtained", "Returned", "Initial size", "Max active", "Max Idle", "Min Idle"];
            }

            List<Object> getValues() {
                String[] connectionPools = mbeanServer.queryNames(hippoPoolingQuery, null)
                def connectionModules = connectionPools.findAll { name ->
                    name.contains('type=PoolingRepositoryMonitor')
                }.collect { new GroovyMBean(mbeanServer, it) }

                return connectionModules.collect {[
                            it.name().getKeyProperty("name"),
                            it.PoolType,
                            it.NumSessionsActive,
                            it.NumSessionsIdle,
                            it.NumSessionsObtained,
                            it.NumSessionsReturned,
                            it.InitialSize,
                            it.MaxActive,
                            it.MaxIdle,
                            it.MinIdle
                ]}
            }

            String getName() {
                return "Hippo connection pools";
            }
        }
        return monitorGroup
    }

Now we need to handle an incoming request, so we need a controller. And in the end we want to show the data on the screen. That is done using a jsp. The following two code blocks show the controller and the important part of the jsp.

@Controller
public class JmxController {
    private GenericMonitor monitor;

    @Autowired
    public JmxController(GenericMonitor monitor) {
        this.monitor = monitor;
    }

    @RequestMapping(value = "/spring/jmx.do", method = RequestMethod.GET)
    public String adminView(ModelMap modelMap) {
        List<MonitorGroup> list = monitor.monitorGroups();
        modelMap.addAttribute("groups", list);

        return "admin/jmx";
    }
}

<c:forEach var="group" items="${groups}">
    <div class="block-error">
        <h2><c:out value="${group.name}"/></h2>
        <table>
            <tr>
                <c:forEach var="columnHeader" items="${group.columns}">
                    <th><c:out value="${columnHeader}"/></th>
                </c:forEach>
            </tr>
            <c:forEach var="valueRow" items="${group.values}">
                <tr>
                    <c:forEach var="valueItem" items="${valueRow}">
                        <td><c:out value="${valueItem}"/></td>
                    </c:forEach>
                </tr>
            </c:forEach>
        </table>
    </div>
</c:forEach>

Of course you are now very curious what we have, well have a look at the following screendump.

Still here? Want more? Ok than we will have a look at the last part of this blog post. The groovy script to read the jmx data from a remote machine. The server side is fine, we can use that now. We have used spring to expose jmx beans over jmxmp. The implementation looks a lot like the JmxMonitor bean that has been discussed before. It should be easy to understand. The code block shows the implementation and after that the output.

def hippoPoolingQuery = new ObjectName('nl.rijksoverheid.importer.repository.pooling:*')
String[] connectionPools = server.queryNames(hippoPoolingQuery, null)

def connectionModules = connectionPools.findAll { name ->
    name.contains('type=PoolingRepositoryMonitor')
}.collect { new GroovyMBean(server, it) }

println "**Found $connectionModules.size hippo connection pools"
if (connectionModules.size) println "Name                     Type      Active Idle Obtained Returned Max Active Max Idle Min Idle Initial Created Destroyed Activated Passivated"
connectionModules.each {m->
    def nameOfEqualLength = m.name().getKeyProperty("name")
    def nameLength = nameOfEqualLength.size()
    if (nameLength < 50) {
        nameOfEqualLength = nameOfEqualLength.padRight (25)
    }
    print m.name().getKeyProperty("name").padRight(25)
    print m.PoolType.padRight(12)
    print "$m.NumSessionsActive".padRight(7)
    print "$m.NumSessionsIdle".padRight(7)
    print "$m.NumSessionsObtained".padRight(7)
    print "$m.NumSessionsReturned".padRight(10)
    print "".padRight(37)
    print "$m.MaxActive".padRight(10)
    print "$m.MaxIdle".padRight(10)
    print "$m.MinIdle".padRight(10)
    print "$m.InitialSize".padRight(12)
    print "$m.NumItemsCreated".padRight(10)
    print "$m.NumItemsDestroyed".padRight(10)
    print "$m.NumItemsActivated".padRight(10)
    print "$m.NumItemsPassivated".padRight(12)
    print "\n"
}

**Found 5 hippo connection pools
Name                     Type      Active Idle Obtained Returned Max Active Max Idle Min Idle Initial Created Destroyed Activated Passivated
writeableJmxRepository   writeable   0      0      0      0         100       5         0         0           0         0         0         0
binariesJmxRepository    binaries    0      0      0      0         100       10        0         0           0         0         0         0
hstConfigJmxRepository   hstConfig   1      1      3      2         25        5         0         0           2         0         3         2
defaultJmxRepository     readonly    0      0      0      0         100       25        0         0           0         0         0         0
previewJmxRepository     preview     0      0      0      0         100       5         0         0           0         0         0         0           

To get started, you can embed the configuration of a complete running environment in a Spring configuration file. Here is the example we’ll use for the remainder of this blog post:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:amq="http://activemq.apache.org/schema/core"
       xmlns:jms="http://www.springframework.org/schema/jms"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
           http://www.springframework.org/schema/jms http://www.springframework.org/schema/jms/spring-jms-2.5.xsd">
  <amq:broker brokername="test-broker" start="true">
    <amq:persistenceAdapter>
      <amq:amqPersistenceAdapter directory="/opt/activemq" maxFileLength="32mb"/>
    </amq:persistenceAdapter>
    <amq:transportconnectors>
      <amq:transportconnector uri="tcp://localhost:7171"/>
    </amq:transportconnectors>
  </amq:broker>
  <amq:connectionFactory id="amqConnectionFactory"  brokerURL="vm://test-broker"/>
  <bean class="org.springframework.jms.connection.CachingConnectionFactory" id="connectionFactory">
    <constructor-arg ref="amqConnectionFactory"/>
    <property name="sessionCacheSize" value="100"/>
  </bean>
  <amq:queue physicalName="testQueue" />
  <bean class="org.springframework.jms.core.JmsTemplate"  id="jmsTemplate">
    <constructor-arg ref="connectionFactory"/>
  </bean>
  <bean class="packageName.ClassName" id="queueProducer">
    <property name="jmsTemplate" ref="jmsTemplate"/>
    <property name="queueName" value="testQueue"/>
  </bean>
  <bean class="packageName.ClassName2" id="queueListener"/>
  <jms:listener-container concurrency="10" connectionfactory="connectionFactory">
    <jms:listener destination="testQueue" ref="queueListener"/>
  </jms:listener-container>
</beans>

That’s a lot of configuration, but let’s analyze each one:

<amq:broker brokername="test-broker" start="true">
  <amq:persistenceAdapter>
    <amq:amqPersistenceAdapter directory="/opt/activemq" maxFileLength="32mb"/>
  </amq:persistenceAdapter>
  <amq:transportconnectors>
    <amq:transportconnector uri="tcp://localhost:7171"/>
  </amq:transportconnectors>
</amq:broker>

This is the definition for an embedded JMS broker, called test-broker which listens on port 7171 using the tcp protocol and which persist data using the default AMQ Message Store in the /opt/activemq directory.

<amq:connectionFactory id="amqConnectionFactory" brokerURL="vm://test-broker"/>

The definition of the JMS connection factory which connects to the broker using a VM transport. In this way the communication is made at the JVM level, thus avoiding network overhead.

<amq:queue id="testQueue" physicalname="TestQueue">

Defines the queue that we are going to use.

<bean class="org.springframework.jms.connection.CachingConnectionFactory" id="connectionFactory">
   <constructor-arg ref="amqConnectionFactory"/>
   <property name="sessionCacheSize" value="100"/>
</bean>

This is the connection factory we are really going to use in our application. It caches connections, sessions and even the MessageProducer. It is important to set the value of the sessionCacheSize property since the default value is 1.

<bean class="org.springframework.jms.core.JmsTemplate" id="jmsTemplate">
   <constructor-arg ref="connectionFactory"/>
</bean>

For those who know the Spring support for JDBC and Hibernate, the JmsTemplate class should sound familiar since it has a very similar design to the JdbcTemplate and HibernateTemplate class.
It hides most of the JMS-related low-level details (e.g. obtaining a session or handling acknowledgments). By default session are not transactional and auto-acknowledged.
The connectionFactory is passed as a property to the constructor.
One of its most important method on the JmsTemplate is send(String destinationName, MessageCreator messageCreator). The MessageCreator is an interface defined by Spring having only the createMessage(Session session) method. Its output is the JMS message that will be sent to destinationName.

<bean class="packageName.ClassName" id="queueProducer">
    <property name="jmsTemplate" ref="jmsTemplate"></property>
    <property name="queueName" value="testQueue"></property>
</bean>

This, finally is an object that is defined within our application, the object that produces the message queue and sends messages to it through the specified JmsTemplate.

<bean class="packageName.ClassName2" id="queueListener"/>

The bean definition that makes up the consumer of our message queue. It needs to implement the javax.jms.MessageListener interface (or some Spring specific interface like SessionAwareMessageListener).
In Spring terms it is a Message Driven Pojo, the difference with a standard Message Driven Bean is that it does not need to run in an EJB container.

<jms:listener-container concurrency="1" connection-factory="connectionFactory">
    <jms:listener destination="AggregateQueue" ref="queueListener"></jms:listener>
</jms:listener-container>

This is the bean who makes the magic happen: it registers itself as a listener for all the queues and executes the callback on the listener as a message is received in an asynchronous way using a Spring TaskExecutor. The concurrency attribute defines how many MessageConsumer instances will be created for each listener. If you use a value > 1 remember to set the prefetchLimit to 1 in order to avoid undelivered messages to the consumers.

Have fun

Test fixtures are now part of the axon-test module. These fixtures allow you to test your command handling logic based on events and commands. You define these tests in a given-when-then style: given these past events, when I execute this command, I expect these events to occur.

The CommandBus interface now allows for asynchronous command dispatching. Instead of using a return value, a Callback can be given to the command bus if you wish to be notified of the result of command execution.

The repositories make use of a UnitOfWork instance, which keeps track of changed aggregates and generated events. Instead of saving each aggregate separately, the UnitOfWork can be committed, causing changed aggregates to be stored, and events to be dispatched. With the new SimpleUnitOfWorkInterceptor or the SpringTransactionalInterceptor, it is no longer necessary to call repository.save(). In fact, you are discouraged from doing so.

Repositories are now capable of detecting conflicting modifications. When loading an aggregate, you may pass in the expected version of the aggregate. An exception is raised if that version does not match the actual version.

Event sourcing repositories allow for a more advanced way of detecting conflicts. Sometimes, not all concurrent modification are in conflict with each other. One user may very well change the shipping address of an order, while another user is adding items. Other modifications, however, are clearly conflicting, such as one user marking an order as shipped, and another one adding a product. For this purpose, you can register a ConflictResolver with the repository. This conflict resolver can, based on unseen and newly applied events, decide whether the changes made by the second user are accepted.

Axon now provides a mechanism to trigger snapshot creation based on the number of events needed to load to reconstitute an aggregate. When the number of events exceeds a given threshold, a snapshot is created and stored in the Event Store.

The SimpleCommandBus and SimpleEventBus now expose statistics over JMX. Although currently quite limited, the statistics will be improved in the coming versions.

All of these features help making applications with CQRS-based architectures even easier. But we’re not there yet. The coming months, we will try to make it even easier with features like Event versioning, support for aggregates with multiple entities, and whatever comes to mind in Axon’s growing community.

For more information about the Axon Framework, see www.axonframework.org.

FTP Client Factory

As the FTP extension for Spring Integration has no official release yet, I have used the latest build which can be found at the Spring Integration Adapters site.

The extension provides a client factory that allows you to connect with a client. The class I have used is the DefaultFTPClientFactory which implements the interface FTPClientFactory.

When you have copied these files to your own project, you can configure a client by using the following code:

    <bean id="defaultClient" class="nl.jteam.importer.ftp.DefaultFTPClientFactory">
        <property value="${ftp.remotedir}" name="remoteWorkingDirectory"></property>
        <property value="${ftp.username}" name="username"></property>
        <property value="${ftp.password}" name="password"></property>
        <property value="${ftp.port}" name="port"></property>
        <property value="${ftp.host}" name="host"></property>
    </bean>

Reading the files

Now that the client is configured, we can read the files from the FTP server. In the method getFilesFromFTPClient() we get a FTPClient by calling the getClient() method on the client factory. The client API provides you with the possibility to retrieve, delete, rename or store files. The API offers a lot more, but I won’t discuss all the methods here. In our case we wanted to retrieve only zip files. As the client does not provide you the functionality to retrieve files from a specific extension, you have to do it yourself by, for example, checking the extension of each file.

Don’t forget to close the connection with the FTP client once you are done with handling files.

Because we want use Spring Integration to send the files to the class that handles the zip files, we create a Message with the list of zip files.

The (partial) code of the FTPInboundAdapter class:

    public FTPInboundAdapter(FTPClientFactory clientFactory, String localTmpDirName) throws IOException {
        Assert.notNull(localTmpDirName, "The directory name to write the files to can not be null");
        this.clientFactory = clientFactory;
        localDirectory = ImportUtils.ensureTempDirExists(localTmpDirName);
    }

    public Message<list><file>&gt; getFilesFromFTPClient() {
        FTPClient client = null;
        try {
            client = clientFactory.getClient();

            List<file> localZipFiles = retrieveRemoteZipFiles(client);

            return MessageBuilder.withPayload(localZipFiles).build();
        } catch (IOException e) {
            throw new MessagingException("Problem occurred while trying to retrieve files.", e);
        } finally {
            closeFtpClient(client);
        }
    }

    private void closeFtpClient(FTPClient client) {
        if (client != null &amp;&amp; client.isConnected()) {
            try {
                client.disconnect();
            } catch (IOException e) {
                logger.warn("Error occurred when disconnection FTP client", e);
            }
        }
    }

The configuration code for the adapter:

    <bean id="ftpInboundAdapter" class="nl.jteam.importer.ftp.FTPInboundAdapter">
        <constructor-arg ref="defaultClient" />
        <constructor-arg value="someLocalDirectory" />
    </bean>

Checking the FTP directory

If you want Spring Integration to check the FTP server on a regular base for new files, you can wire up an inbound channel adapter with a cron expression like this:

    <si:inbound-channel-adapter id="zipInboundChannelAdapter" ref="ftpInboundAdapter" method="getFilesFromFTPClient" channel="zipFilesChannel">
        <si:poller max-messages-per-poll="1">
            <si:cron-trigger expression="0 0/5 * ? * *" />
        </si:poller>
    </si:inbound-channel-adapter>

The channel attribute of the inbound channel adapter specifies the output channel. So in our case this will be the message with the list of zip files that is put onto this channel. This channel can then be used to send the message to wherever you want.

As you could see, it was relatively easy to connect with a FTP server and retrieve the files. I hope this post helped you in setting up your own FTP connection with Spring Integration. All that rests us is waiting for official release of the Spring Integration Adapters.

Maven dependency

If you use Maven, you can simply add the following dependency to use the library.

<dependency>
    <groupid>net.htmlparser.jericho</groupid>
    <artifactid>jericho-html</artifactid>
    <version>3.1</version>
</dependency>

API

I don’t want to explain all classes, but the following classes are basically the starting point of all your parsing.

• Source – Represents a source HTML document. This is always the first step in parsing an HTML document.
• OutputDocument – Represents a modified version of an original Source document or Segment.
• Element – Represents an element  in a specific source document, which encompasses a start tag, an optional end tag and all content  in between.

For a complete overview of all classes you can view the javadoc.

Extract all text

To extract all the text from the HTML markup, all you have to do is the following:

    public String extractAllText(String htmlText){
        Source source = new Source(htmlText);
        return source.getTextExtractor().toString();
    }

You define a new Source object that takes in our case a String as input. But it also accepts for example a InputStream or URL. The Source object contains a method getTextExtractor that allows you to, how surprising, extract the text. The TextExtractor class gives you a few options to configure the extraction. One of the options is that you can exclude text from a specified Element. You can also include an attribute. The value of that attribute will be included in the output.

Manipulating HTML

Manipulating HTML is very easy with Jericho. In the code example below I want to add an id attribute to all H2 elements to create anchor navigation. One again I create a Source document. From this Source document I create an OutputDocument.

The OutputDocument represents a modified version of the original Source document. With the list of all H2 elements retrieved from the Source, we now can ask for all the attributes of a single H2 element. If the attribute id already exists we do nothing, but if it does not we recreate the starttag with a new id attribute and all the other existing attributes from that H2 element.

As you can see in the example, it is relatively easy to manipulate attributes of an element. With the Attributes object you can get a List of Attribute objects that are found in the source document or in a starttag. These attributes are not modifiable. The outputDocument has a convenience method that allows us to replace the specific startTag with our newly created H2 start tag in order to add our id attribute.

    public String addIdAttributeToH2Elements(String html) {
        Source source = new Source(html);
        OutputDocument outputDocument = new OutputDocument(source);
        List<element> h2Elements = source.getAllElements("h2");

        for (Element element : h2Elements) {
            StartTag startTag = element.getStartTag();
            Attributes attributes = startTag.getAttributes();
            Attribute idAttribute = attributes.get("id");

            if (idAttribute == null) {
                String elementValue = element.getTextExtractor().toString();
                String validAnchorId = AnchorUtils.getLowerCasedValidAnchorTitle(elementValue);

                StringBuilder builder = new StringBuilder();
                builder.append("<h2").append(" ").append("id=\"").append(validAnchorId).append("\"");
                for (Attribute attribute : attributes) {
                    builder.append(" ");
                    builder.append(attribute);
                }
                builder.append(">");

                outputDocument.replace(startTag, builder);
            }
        }

        return outputDocument.toString();
    }

Remove Elements

Just like me, you may want to remove a few tags from your HTML. Here is an example that shows you how you can achieve that.

    private static final Set<string> ALLOWED_HTML_TAGS = new HashSet<string>(Arrays.asList(
            HTMLElementName.ABBR,
            HTMLElementName.ACRONYM,
            HTMLElementName.SPAN,
            HTMLElementName.SUB,
            HTMLElementName.SUP)
    );

    private static String removeNotAllowedTags(String htmlFragment) {
        Source source = new Source(htmlFragment);
        OutputDocument outputDocument = new OutputDocument(source);
        List<element> elements = source.getAllElements();

        for (Element element : elements) {
            if (!ALLOWED_HTML_TAGS.contains(element.getName())) {
                outputDocument.remove(element.getStartTag());
                if (!element.getStartTag().isSyntacticalEmptyElementTag()) {
                    outputDocument.remove(element.getEndTag());
                }
            }
        }

        return outputDocument.toString();
    }

In the example above you see that after checking if the tag is allowed, we need to remove the start and endtag. If you would remove the complete element, then you would also remove the text within these tags. The API allows you to check for elements that are empty. This can be handy to remove redundant empty elements or in my case to check if the starttag a self closing tag.

Conclusion

In this post I showed you how I have used Jericho, but Jericho has a lot more interesting features. On their webpage they provide more examples on how to use those features. Jericho provides a nice and clean API and makes the parsing of HTML really easy!

Estimators for recommendations

Let’s start with the main usage of estimators: providing recommendations. Suppose we create a GenericItemBasedRecommender, provide it with a DataModel and one of Taste’s ItemSimilarity implementations.

To fetch a few recommendations we call GenericItemBasedRecommender.mostSimilarItems(long itemID, int howMany), as shown in the snippet below:

  @Override
  public List<RecommendedItem> mostSimilarItems(long itemID, int howMany) throws TasteException {
    return mostSimilarItems(itemID, howMany, null);
  }

  @Override
  public List<RecommendedItem> mostSimilarItems(long itemID, int howMany,
                                                Rescorer<LongPair> rescorer) throws TasteException {
    TopItems.Estimator<Long> estimator = new MostSimilarEstimator(itemID, similarity, rescorer);
    return doMostSimilarItems(new long[] {itemID}, howMany, estimator);
  }

After delegating the method call to a more generic mostSimilarItems method, a MostSimilarEstimator is constructed and passed to the protected method doMostSimilarItems. The whole process of estimating and recommending is implemented via an estimator and algorithm specific logic within a recommender.

Now let’s zoom in on the doMostSimilarItems method. See the snippet below:

  private List<RecommendedItem> doMostSimilarItems(long[] itemIDs,
                                                   int howMany,
                                                   TopItems.Estimator<Long> estimator) throws TasteException {
    DataModel model = getDataModel();
    FastIDSet possibleItemsIDs = new FastIDSet();
    for (long itemID : itemIDs) {
      PreferenceArray prefs = model.getPreferencesForItem(itemID);
      int size = prefs.length();
      for (int i = 0; i < size; i++) {
        long userID = prefs.get(i).getUserID();
        possibleItemsIDs.addAll(model.getItemIDsFromUser(userID));
      }
    }
    possibleItemsIDs.removeAll(itemIDs);
    return TopItems.getTopItems(howMany, possibleItemsIDs.iterator(), null, estimator);
  }

The snippet above describes the core logic for item-based recommendation. This process consists of three steps:

1. Fetch all preferences for the given item(s)
2. For each preference get the corresponding user and fetch all their other preferences
3. From this set of preferences, minus the given item, get the corresponding items and determine the top items based on the given estimator

The TopItems is a helper class for fetching the top ranked items of a set of items for a given estimator.

Now on to the estimator. All estimators implement TopItems.Estimator<T> interface which is really simple. It returns an estimate for a ‘thing’ as a double.

  public interface Estimator<T> {
    double estimate(T thing) throws TasteException;
  }

Now on to the MostSimilarEstimator:

  public static class MostSimilarEstimator implements TopItems.Estimator<Long> {

    private final long toItemID;
    private final ItemSimilarity similarity;
    private final Rescorer<LongPair> rescorer;

    public MostSimilarEstimator(long toItemID, ItemSimilarity similarity, Rescorer<LongPair> rescorer) {
      this.toItemID = toItemID;
      this.similarity = similarity;
      this.rescorer = rescorer;
    }

    @Override
    public double estimate(Long itemID) throws TasteException {
      LongPair pair = new LongPair(toItemID, itemID);
      if ((rescorer != null) && rescorer.isFiltered(pair)) {
        return Double.NaN;
      }
      double originalEstimate = similarity.itemSimilarity(toItemID, itemID);
      return rescorer == null ? originalEstimate : rescorer.rescore(pair, originalEstimate);
    }
  }

This estimator does three things:

1. Use the Rescorer to filter items. Rescorers can be used to create domain specific filtering of items
2. Use the ItemSimilarity to calculate the preference of a user for the given item
3. Optionally boost the similarity value with the Rescorer

This setup allows you to plugin arbitrary ItemSimilarity algorithms in the recommender.

Recommended because…

Another interesting feature of the GenericItemBasedRecommender is the ‘Recommended because’ feature. With this feature you can determine why a certain item was recommended to you, i.e. which of your preferences were largely responsible for giving you this recommendation.

To use this feature call recommendedBecause(long userID, long itemID, int howMany), see snippet below:

  @Override
  public List<RecommendedItem> recommendedBecause(long userID, long itemID, int howMany) throws TasteException {
    if (howMany < 1) {
      throw new IllegalArgumentException("howMany must be at least 1");
    }

    DataModel model = getDataModel();
    TopItems.Estimator<Long> estimator = new RecommendedBecauseEstimator(userID, itemID, similarity);

    PreferenceArray prefs = model.getPreferencesFromUser(userID);
    int size = prefs.length();
    FastIDSet allUserItems = new FastIDSet(size);
    for (int i = 0; i < size; i++) {
      allUserItems.add(prefs.getItemID(i));
    }
    allUserItems.remove(itemID);

    return TopItems.getTopItems(howMany, allUserItems.iterator(), null, estimator);
  }

It takes all items the given user has a preferences for, minus the given item and passes this to TopItems, along the with RecommendedBecauseEstimator, see the code below:

  private class RecommendedBecauseEstimator implements TopItems.Estimator<Long> {

    private final long userID;
    private final long recommendedItemID;
    private final ItemSimilarity similarity;

    private RecommendedBecauseEstimator(long userID, long recommendedItemID, ItemSimilarity similarity) {
      this.userID = userID;
      this.recommendedItemID = recommendedItemID;
      this.similarity = similarity;
    }

    @Override
    public double estimate(Long itemID) throws TasteException {
      Float pref = getDataModel().getPreferenceValue(userID, itemID);
      if (pref == null) {
        return Float.NaN;
      }
      double similarityValue = similarity.itemSimilarity(recommendedItemID, itemID);
      return (1.0 + similarityValue) * pref;
    }
  }

}

This RecommendedBecauseEstimator determines the ranking by multiplying the preference value of the user by the item similarity value of the current item pair. After this process the top ranked items are those items that were most important in causing a recommendation of the given item.

Conclusions

This concludes the overview of some Taste internals and has hopefully given you a clearer picture on how recommendations and estimators work inside Taste. In future posts I will probably expand on this topic, especially within the context the evaluation of recommenders. If you have any questions regarding Taste in general or this topic of estimators feel free to leave a comment.

If you’re one of those attendees or you missed the presentation, you can download the slides here:

• Lucene Eurocon Slides
• Berlin Buzzwords Slides

At Lucene Eurocon, the first European conference on Lucene and Solr there were interesting presentations, ranging from practical relevance to language analysis. For me it was fun to give a practical presentation about recommendations as a complementary feature to search applications. I hope you find the presentation useful if you’re trying to work out how to build a recommender – I used the movielens dataset as an example in the presentation and based the code on my earlier ‘getting started’ blog.

I also really enjoyed doing the Berlin Buzzwords presentation and meeting up with people from the Mahout community and other attendees. This conference focused mainly on NoSQL, scalability and Hadoop. However, from my talks with people there I sense that there’s growing interest in Mahout. You should find the presentation useful if you want to know more about different algorithms and how to evaluate them. I will blog about this topic in more detail soon.

Until then, I’d love to hear some feedback on what you think of the presentations!

In this post I will show you my selection of apps that I found useful, interesting or just fun to have.

The useful apps

9292ov Pro

This application allows you to plan your trip with all Dutch public transport. It also shows you how much you must pay for the tickets and you can share your advise via mail or Twitter. Furthermore, it holds a list of all delays, allows you to save your advise or explore places of departure or arrival locations by using the map function. And for the people that can not read Dutch, it’s also available in English.

ES File Explorer

There are a lot of file managers in the Market, but I chose this one, because it has a good interface that doesn’t hurt my eyes and has a few interesting features. It has support for FTP and allows you to manage/(un)install your apps. Beside managing your files on the phone, you can also manage your files on a PC via LAN.

Task Manager

Android keep your application open at the background. This is really nice, but also can make you phone slow or maybe even worse, drain your battery. Task Manager allows you to close these applications. Just like the file manager, there are a lot of these apps. I think that this one, is a nice and simple application, that is easy to understand and easy to use.

JuiceDefender

The batteries of smart phones in general don’t last really long. Juice Defender is an app that saves power and extends your battery life by controlling the data connection and WiFi of the device. You can schedule the activation of the data connection or even turn it off. I have scheduled it to enable my data connection for 1 minute every 15 minutes. You also have an ultimate (paid) version, which give you more possibilities.

ConnectBot

Every developer wants to have a SSH client on this mobile phone . ConnectBot is one of the available clients in the Market. It can manage simultaneous SSH sessions, copy/paste between other applications and even create secure tunnels.

The show off apps

Google Goggles

With Google Goggles you can search for an object with your camera phone. It can identify text, landmarks, books, contact info, artwork, places. wine and logos. With your camera you make a picture, which will be analyzed. If the object is recognized, the app will provide you with the found information.  It works quite good and is really a must have app on your android phone.

Google Sky Maps

Google Sky Maps lets you discover and browse the night sky. It uses your phone’s orientation sensors to show you a star map for your location. You can search for planets, stars, constellations and the application will point you to the right direction. Since version 1.5 they added image of Hubble.

Layer Reality Browser

Layar is an augmented reality browser that, as they say it, shows you the things you can’t see. The app is provided with hundreds of layers (free and paid) that allows you to find the most useful things as hotels and restaurants or the most crazy things as your Avatar’s body. The Layar Catalog contains a lot of local and international layers. You (or JTeam) can even create your own layer. Really cool and also useful for finding things nearby.

The fun apps

Ringdroid

Ringdroid is a ringtone creator. This app lets you manipulate any song of your phone. After you have selected a song you can easily select a part of the song via two sliders. If you have selected the right part, you can save the song. Ringdroid then gives you the option to use the song as a ringtone and/or a notification.

Abduction

Abduction is an addictive game for your Android phone. Your herd has been abducted by aliens, so now you have to save your friends. It uses the accelerometer, meaning you have to tilt your phone to make your way up to the aliens. To get to the spaceship you have to bounce up in a series of platforms. It has nice graphics and it is a good time-wasting game.

Epic Fails

The term “epic fail” is popular on the Internet. People place it onto photos or short videos that depict unsuccessful events or people falling short of expectations. Epic Fails shows you the most hilarious fail pictures. Nothing more, but extremely funny to show to your friends. You can share the pictures with your friends via all kind of applications like mail, facebook or picasa.

PicSay

PicSay is a photo editor. You can add effects, word-balloons, titles and graphics. The pro (paid) version contains a lot more features and is propably more worthwhile. It has an intuitive interface, which is easy to use. I had a little bit of trouble to resize the items, but you probably just have to get used to it.

JTeam and Android

Within JTeam we have the knowledge and experience to create Android applications. So if you have a great idea for an application or maybe want to have a mobile app version of your website, you can contact us by filling in the contact form here and we can help you conquer the Android world.

JTeam today announced that it has partnered with New Relic, Inc., to make New Relic RPM available to its clients during consulting engagements. JTeam consultants and clients can now use RPM collaboratively to monitor, troubleshoot, and optimize live production web applications and Solr instances, and thus ensure successful application deployments and a superior Web user experience.

New Relic, Inc. is the leading software-as-a-service provider of application performance management (APM) solutions. Its flagship product, New Relic RPM, is an on-demand performance management solution for web applications developed in Java, Ruby, or JRuby and provides deep, 24×7 visibility and code-level diagnostics for web applications deployed on traditional, dedicated infrastructures or in the cloud. New Relic recently announced RPM’s ability to provide deep visibility into production Solr instances. To learn more about New Relic RPM and Solr monitoring go to http://www.newrelic.com/solr.html.