JTeam today announced that it has partnered with New Relic, Inc., to make New Relic RPM available to its clients during consulting engagements. JTeam consultants and clients can now use RPM collaboratively to monitor, troubleshoot, and optimize live production web applications and Solr instances, and thus ensure successful application deployments and a superior Web user experience.

New Relic, Inc. is the leading software-as-a-service provider of application performance management (APM) solutions. Its flagship product, New Relic RPM, is an on-demand performance management solution for web applications developed in Java, Ruby, or JRuby and provides deep, 24×7 visibility and code-level diagnostics for web applications deployed on traditional, dedicated infrastructures or in the cloud. New Relic recently announced RPM’s ability to provide deep visibility into production Solr instances. To learn more about New Relic RPM and Solr monitoring go to http://www.newrelic.com/solr.html.

The Lucene Connectors Framework, an incubator project at Apache, provides a framework for connecting a source content repository to target repositories or indexes, such as Apache Solr. Last month the Lucene Connector Framework published their first build-able sources.

Basic Introduction to LCF

The Lucene Connector Framework has been developed and successfully deployed for the last five years by MetaCarta. Giving us a proven and usable system, but with a rather steep learning curve. There is a lot of documentation already available on the project’s wiki and for this blog post we will take a look at how to build and deploy LCF.

The Lucene Connectors Framework consists of three main components:

• Configuration UI (Web Application)
• Authority Service (Web Application)
• Agent (Java Process)

Within LCF, the Agent process does the actual work, it crawls documents and ingests them. The Authority Service webapp is used to get authorization tokens for a given username. To configure and interact with the system, the Configuration UI is used. The Configuration UI (crawler-ui) is a webapp in which you can configure the system, start/pause/abort the crawler and get statistics about the documents.
The agent, authority service and configuration ui, are build on top of the PostgreSql database. The database is used to store state, eg. crawled documents, and configuration, eg. scheduling and connection information. The synchronization folder is used to keep the processes in sync, by providing a locking mechanism across jvm instances.

Build

There are currently no releases of LCF available, but the project sources are in svn. Get the project sources using subversion from:
https://svn.apache.org/repos/asf/incubator/lcf/trunk

After a checkout, you will find the following directories:

• documentation
  • Part of the end-user documentation, written in LaTeX.
• modules
  • This directory contains the sources for the framework and the different supported connectors.
• tests
  • Part of the test files used by MetaCarta

Building LCF is done using ant and requires ant 1.7 or greater. Run ant from the modules directory to build the system. The output is produced in the modules/dist folder. The build creates two war files in the tomcat folder and a set of jar files in the processes folder.

Setup

The LCF is implemented on top of a postgreSQL database and uses a folder on the file system for synchronization. Both need to be configured in the LCF configuration file. By default LCF looks for a configuration file at /lcf/properties.ini. The following code block shows the minimal configuration needed.

org.apache.lcf.synchdirectory=/home/user/lcf_work/
org.apache.lcf.database.username=postgres
org.apache.lcf.database.password=postgres
org.apache.lcf.database.name=lcf

When not specified in the configuration file, LCF expects a standard commons-logging property file at /lcf/logging.ini.

org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger

# Configuration file of the log
log4j.configuration=log4j.xml
log4j.rootCategory=info
# Set root logger level to DEBUG and its only appender to A1.
log4j.rootLogger=DEBUG, A1

# A1 is set to be a ConsoleAppender.
log4j.appender.A1=org.apache.log4j.FileAppender
log4j.appender.A1.Threshold=DEBUG
log4j.appender.A1.File=ApplicationLog.log
log4j.appender.A1.Append=true

# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x – %m%n

Create the database and register the different components using the following commands from the modules/dist directory.

java -Djava.ext.dirs=processes org.apache.lcf.core.DBCreate postgres postgres
java -Djava.ext.dirs=processes org.apache.lcf.agents.Install
java -Djava.ext.dirs=processes org.apache.lcf.agents.Register org.apache.lcf.crawler.system.CrawlerAgent
java -Djava.ext.dirs=processes org.apache.lcf.agents.RegisterOutput org.apache.lcf.agents.output.solr.SolrConnector “Solr Connector”
java -Djava.ext.dirs=processes org.apache.lcf.crawler.Register org.apache.lcf.crawler.connectors.webcrawler.WebcrawlerConnector “Web Crawler”

Run

With the configuration in place you need to deploy the lcf-crawler-ui web application and start the Agent. The web application can be deployed in any servlet container like Tomcat or Jetty, but needs to be run as the same user as the Agent. You can find the war file needed in the modules/dist/web directory.

The Agent is started by using the following command from the modules/dist directory.

java -Djava.ext.dirs=dist/processes org.apache.lcf.agents.AgentRun&

Stopping the agent is done using the AgentStop class. Don’t stop the Agent in another way, because that may result in dangling locks in the synchronization directory.

The lcf-crawler-ui should now be working at http://hostname/lcf-crawler-ui

Configure

Using the lcf-crawler-ui webapplication you can configure the web-crawler. The web-crawler needs an output connection, a repository connection and a job. The authority connection will not be used by the webcrawler.

Create an output connection of the type “Solr Connector” and specify the details of your solr installation. The update handler defaults to /update/extract, as the Solr connector uses the extraction handler to send the data. Make sure the Solr extraction handler is correctly configured in your solr installation.

Create a repository connection of the type “Web Crawler” and at least specify an Email address. This email address is send in the head of every request and can be used by system administrators to contact you about the
crawling.

Create a Job for the crawling the web by selecting the two created connections. The web-crawler uses the Seeds you specify as its starting point, so you need to specify at least one url.

Crawl

You can then look at the status of your job by clicking “Status and Job Management” on the sidebar. You can start any crawl you like immediately from this interface by clicking “Start” next to the name of the job. This interface also allows you to see how many documents have been crawled.

Conclusion

The Lucene Connectors Framework provides a solid and generic infrastructure for connecting source repositories, to target repositories or indexes. Build on top of postgresql makes it robust and rigid against system crashes or restarts.

Of course there is still a lot of work to be done. The Lucene/LCF authorization model has yet to be worked out and more connectors are needed to support different source and output repositories. Maybe a new ui?

Solr 1.5

With Solr 1.4 released in November ’09, Solr 1.5 carried on where 1.4 left off. Most development was related to bugs that had been found in 1.4, but a number of interesting new features were added:

• Spatial Search Support – Following on from the development of Solr’s spatial search support in SOLR-773, Grant Ingersoll lead a drive to complete Solr’s spatial search support. Providing integration with Spatial Lucene, and introducing its own spatial search functions, Solr’s spatial search allows users to filter their results to those within a certain distance of a point, and boost relevance of results by their distance
• Poly FieldTypes – From the Spatial Solr development, Poly FieldTypes allow one field in the Solr schema to be actually indexed as multiple fields. Although currently only used for spatial search, this functionality promises to bring complex FieldTypes to Solr.
• Sorting by FunctionQueries – Also from the Spatial Sol development, the ability to sort by FunctionQueries allows users Solr’s many FunctionQueries in their sort criteria, as though they were normal fields. This allows users to sort Events by how close their start date is to NOW for example.
• Multi-threaded DataImportHandler – With the DataImportHandler’s (DIH) widespread acceptance, it was only a matter of time before it was pushed to its limits. One improvement to the DIH made in Solr 1.5, was the ability to use multiple threads to request and index Documents in parallel, thus drastically improving the performance of those users indexing vast datasources.

Perhaps Solr 1.5′s biggest limitation was that it was based on Lucene 2.9.x, and consequently was unable to benefit from the many new features being introduced in Lucene 3.x. As part of a wider decision to merge the development of Solr and Lucene, Solr 1.5 was shelved (tagged in SVN) in favour of a new version of Solr that would be built against Lucene’s trunk (more about that later).

At this stage it is hard to tell if Solr 1.5 will be released or not. Certainly if you need one of the features mentioned above, or are facing one of the many bugs fixed in Solr 1.5, it would be worth checking it out from SVN and upgrading, otherwise I recommend staying with Solr 1.4, and waiting for the newer version of Solr:

Solr 3.1

Before carrying on, I should say that it has not been decided that the next version of Solr will be given the version number 3.1. The name comes from the fact that it is built against the current development version of Lucene which will be released as 3.1.

Solr 3.1, as just mentioned, is a considerably updated version of Solr that is built against Lucene’s development trunk and is updated as Lucene is developed further. For users wanting bug-free software, this means that Solr has access to Lucene’s considerable test-suite, and any changes made to Lucene will be tested against Solr. For users wanting bleeding edge features, this means that Solr will be updated shortly after a feature is added to Lucene.

At the time of writing, this version of Solr is still under heavy development and should be treated as an alpha version at best. The task of updating Solr to match Lucene meant a considerable number of changes had to be made, and many developments such as adding per-segment facetation still need to occur.

With some of Lucene’s latest features (namely flexible indexing) needing to ‘bake’ in the open for awhile before being released, I imagine it will be some time before Solr 3.1 (or whatever it will be called), will be released.

Solr Cloud

Somewhat parallel to these developments has been the effort to add ‘cloud’ support to Solr using Apache Zookeeper. Solr Cloud, currently residing as a branch, provides support for automatic node detection and index/configuration sharing. The intention is to allow users to be able to deploy Solr in their cloud, and be able easily add and remove nodes.

At time of writing, steps are being taken to bring the Solr Cloud branch back into trunk. However, it was noted by one of the developers, that it does not include all the features that they wish to add.

What version to use?

With all the changes currently occurring with Solr, I would go so far as to say that users should continue to use Solr 1.4. However, if you need access to one of the many new features introduced in Solr 1.5+ or Lucene 3.x, then given Solr 3.1 a shot, and report back your experiences.

Enterprise Search

Enterprise Search is the application of Information Retrieval (IR) technology to the information within an organization (as opposed to Web Search and Desktop Search). This can be all sorts of data (and usually a combination of multiple sorts), for example products in a database, documents on a file system and emails from a company mail server. This typically includes several distinct steps: gather content, index the retrieved content and make the content searchable.

Gathering content

The first thing to do is make sure the data that needs to be searchable is retrieved somehow from its respective location. This usually means using and/or building connectors that can access the content and retrieve it. There are currently several open source crawling tools out there, but Lucene Connector Framework (LCF) is by far the most promising one. Luckily for us, LCF integrates with Solr out-of-the-box. You can use LCF to crawl a website or intranet site, but also other sources of information (e.g. a database, file system and Sharepoint and many more) and pass all the found documents (before or after extracting the useful content) to Solr.

Extracting data

Once we get to the data we want to make searchable, we usually have to transform it to a format that is understandable by the indexing / search engine. In order to achieve this there are several alternatives. Which one is more suitable for you depends on the specific requirements you have.

The first option is using Solr’s built in support for Apache Tika, a toolkit for extracting metadata and structured text content from various document formats (e.g. Word and PDF documents). This allows you to send the raw document content to Solr and then have Tika extract the right content that is subsequently indexed.

If you need broader support for document formats, there are also several commercial file readers available with the most important being ISYS file readers.

Indexing / Searching

The last part is making sure users can actually search the indexed data sources. The search algorithms and relevance rankings are handled by Solr (with Lucene under the hood) and can be configured and extended to the max. The main advantage of using an open source search solution is that it allows you to easily extend the default functionality with custom logic.
The good thing about using Apache Solr is that it provides a whole lot of functionality out of the box, including facetted navigation, “Did you mean…” and “More like this” functionality. This allows you to easily impress your users with a minimum effort.

Conclusion

As mentioned in the introduction, we at JTeam feel that organisations that are considering using an Enterprise Search solution, should at least consider using Apache Solr and its related technology stack. The main advantage of these is that they are open source, which entails no license fees, but also built on a highly performant, proven technology: Apache Lucene.
If you have are considering Solr, have any questions about using Solr for your project or need help convincing others about using Solr, contact us!

Introduction

To finish the last part for my Bachelor of ICT, I recently started my internship at JTeam. JTeam has been doing a lot of projects using either a CMS (e.g. Magnolia and Hippo) or requiring a search solution, typically using Apache Lucene and Apache Solr. My assignment is to investigate the problem of making the information from different CMS search-able using Solr and hopefully come up with a good solution.

In order for the content in a CMS to be available for searching, it needs to be indexed by Solr. Problem is that many of the Java-based CMS solutions do not provide an easy way to get their content to Solr. We need to get the information out of the CMS and feed it to Solr. In this blog post I will discuss the synchronization problem at a more generic level and look at some possible solutions.

Synchronization problem

Before diving into CMS solutions specifically, let’s look at a more classic problem: the data replication problem. When we get information from a specific source and replicate it into another system, the data may get out of date when changes in the original source are not immediately propagated. This is a common problem, that has been studied extensively. Junghoo Cho and Hector Garcia-Molina studied this problem with the focus on Web Crawlers. They started their study by looking how to measure the problem and defined the freshness and age of a database.

Intuitively, we consider a database ”fresher” when the database has more
up-to-date elements. For instance, when database A has 10 up-to-date elements out of 20 elements, and when database B has 15 up-to-date elements, we consider B to be fresher than A. Also, we have a notion of ”age”: Even if all elements are obsolete, we consider database A ”more current” than B, if A was synchronized 1 day ago, and B was synchronized 1 year ago.

With this notion of Freshness and Age we can look at the potential quality of the solutions to this problem.

Possible ways

Basically, there are three ways of replicating the content in a CMS to a search solution like Apache Solr:

1. Look at the CMS’s data to see what has changed
2. Listen to the CMS and hear what has changed
3. Let the CMS update Solr

The first option, looking at the CMS’s data to see what has changed, can be done by creating a crawler. The crawler will iterate over all the content in the CMS and compare the last known version with the current version of the content in the CMS. Such a crawler is typically implemented by repeating the following steps:

• Put all the known elements in a Queue
• While the Queue is not Empty
• Take an element from the Queue
• Check if the element has changed

This is fairly straightforward to implement. Luckily, the same study shows there is little to gain in trying to optimize the order in which we look at the elements, or the amount of resources we use for specific elements (e.g. the rate at which specific elements are checked). However, in order to keep your Solr data fresh and young, a crawler will need more and more resources when the CMS grows.

The second option, listening for changes in the CMS, would be better. In order to listen for changes the CMS should push the changes to external systems. A push, or observation, mechanism can be implemented in one of two ways: asynchronous or journaled. The simplest form, asynchronous observation, will send a notification on every change that occurs in the CMS. Getting notified of the change and being able to incorporate that change will give us a fresh and young Solr index. Although asynchronous observation looks perfect at first sight, it has two flaws: it doesn’t guarantee that all changes are actually sent and does not guarantee the right ordering of the changes. Missing a change, or getting two updates in the wrong order, leaves an element out-of-date, and the Solr index not fresh.

Journal based observation solves the problems of asynchronous observation, as it can guarantee complete information and ordering. Instead of being informed on every change, we will get a list, a journal, of changes since the last time we checked. Using journal based observation, the age is only influenced by the time it takes to process the changes and the freshness by the amount of changes in each journal.

The third option, let the CMS update Solr, is obviously the most ideal. Instead of crawling or observing the CMS and update Solr when something changes, the CMS could directly propagate all changes to Solr. The age and freshness of Solr would be perfect, as changes to data can be made available at the same time to both the CMS and Solr. Having a CMS that is built using an event-driven, highly decoupled architecture, for instance using a CQRS framework, like Axon Framework, would make extending the CMS with the capability for updating Solr very easy. However, currently no Java based CMS provides this capability out of the box.

Conclusion

Trying to solving the problem of making the information from different CMS solutions search-able using Solr gave some interesting insights in the classic data replication problem. Using Age and Freshness as a guideline, letting the CMS update Solr looks like the best solution, but is currently not usable when looking at different CMS solutions. Both asynchronous observation and crawling have up and downsides, but seem to be the most generic solution. Journaled observation, when looking at the Age and Freshness and different CMS systems, looks the most promising.

In the next blog posts as part of this series, I will go into more detail of actually implementing this integration for a number of widely used CMS solutions like Magnolia and Hippo. Stay tuned…

References

Effective Page Refresh Policies For Web Crawlers
Axon framework

The thing is that both Ms Fast and Endeca did not implement their language analysis themselves. They use an existing, commercial solution called the Rosetta Linguistics Platform (RLP) provided by Basis Technologies under the hood to provide their sophisticated language analysis capabilities. This is a good thing, as RLP also provides integration components for Apache Solr. This allows anyone using Solr to easily plug in RLP advanced language capabilities into their solution.

What is RLP?
The Rosette Linguistics Platform (RLP) is a commercial solution that allows you to perform linguistic analysis of text in many languages (English as well as dozens of major European, Asian, and Middle Eastern languages). Besides that RLP also supports advanced entity extraction capabilities, base noun detection, sentence boundary detection and even part of speech tagging. However, as RLP is a commercial product it comes with a price tag.
However, we feel that RLP has a lot of potential when you are in need of sophisticated language capabilities. That’s why we want to show you how to integrate RLP into your solution based on Apache Solr.

Installing the RLP platform for Solr
RLP and Solr are two separate systems. Integration between Solr and RLP is quite straight forward. Basis provides extensive documentation on how to setup the RLP on your machine. With the RLP configuration you can customize the actual language analysis. This is not done directly via Solr. Installing and configuring RLP is explained in RLP documentation that is included with RLP bundle.

Configuring Solr to use RLP for analyzing your documents is also relatively easy to setup. In order to use RLP’s language analysis in Solr you have to configure the RLPTokenizerFactory as tokenizer for the specific fields you want to use it on. As RLP is not written in Java, JNI is used to integrate a Java application with RLP. In order for Solr to integrate with RLP you must set certain environment variables that point to the RLP installation. Basis has quite extensive documentation for the Solr integration with RLP as well, which is included in the RLP Solr bundle.

RLP’s stemming capabilities
Terms like verbs, nouns and adjectives appear in different forms. When searching the different forms of a term, it can result in a miss. For example if you search for the term customize, but the token in the index is customization, your are unlikely to find results with this term. The solution to this is stemming. Stemming reduces a term to a base form. There are many ways to do this but the most common way is to remove the terms’ suffix according to some basic rules. In our case customize and customization would be stemmed to customiz. Stemming is usually applied during indexing and during searching (stemming the search query). So whether we search for customization or customize does not matter, for both terms we’ll get the same result. Another example is go and going, both forms are stemmed to go.

In all languages there are terms that are irregular. For example the verb to buy. The present tense is buy and the the past tense is bought. Using a suffix based stemmer, will not create a common base form for the tokens. The create a proper base form the stemmer needs to have some basic knowledge of a language. Stemmers who do this usually use lemmatisation. Using this algorithm buy and bought would both be stemmed to buy. Lemmatisation based stemmers will stem more terms to a common base form and therefore increase findability of documents in your index.

RLP’s stemming capabilities fall into the last category of stemming algorithms and as you can see in the table below the stemming is quite powerful.

Advantages of using RLP
Why would you use RLP in conjunction with Solr? RLP provides you with really powerful language analysis. Solr is a extensible open-source search engine. By integrating RLP with Solr, RLP will compliment the language analysis provided by Solr and increase the quality of your search results. As seen in the previous table, the better stemming capabilities increases the likelihood for relevant documents to be found.

If you consider using RLP for your project or solution, please feel free to contact us, so we can help you both to make a decision as well as help you implement it.

• PDF documentation has been improved to remove inconsistencies in request parameter and source code package names
• SpatialFilter now includes hashCode and equals implementations, facilitating storage of the filter in caches

JTeam’s Solr Spatial Plugin (SSP) is a standalone plugin that provides efficient and extensible spatial search support to both Solr and Lucene. You can find more information about the plugin here

So I’m pleased to announce the first Dutch Lucene User Group Meetup. It will take place on 17th February (Wednesday) at the JTeam headquarters office. This first meetup will be split into two parts:

• Introduction to the user group and the members. We’ll have a discussion about what we would all like to see coming out of this user group and what activities we would like to have.
• The next part will be more technical. Anne Veling will share with us some of his experience of large scale Solr deployment that he’s working on.

If you wish to attend, please send us an email to: events@lucene-nl.org

Date: 17th February 2010

As highlighted by Grant Ingersoll, the Solr committer who released the figures, the most voted for feature by far is field collapsing, otherwise known as result grouping, an extension to Solr primarily developed and maintained by my colleague Martijn van Groningen. Field collapsing allows results which have the same value for a certain field to be collapsed into a single result. This can prove useful when you get 1000 variations of the same heater as the result of a query, as is the case for one of our current customers. You can find out more about field collapsing from Martijn’s blog entries.

The second most voted for feature is incorporating Local Lucene/Solr. This extension allows Solr users to conduct location based searches, such as where are pizza shops near my house, through the Spatial Lucene contrib. I personally have been very involved in the development and presentation of this feature, and I have followed up on my initial contributions with a fully refactored version that is available here on the JTeam. We will be contributing these developments, particularly those on the Lucene side, back to the community in a matter of days.

We have used both features in multiple projects and have always received very positive responses. However, both features are still only Jira issues, so if you haven’t already voted for them, please do, so we can ensure that they become part of the next version of Solr.

What do a directory services company, a wholesaler of bathroom fittings,  a social events guide, an oceanographic data centre and the pan-European library initiative have in common? They all need to offer their clients the ability to search and filter results within a flexible geographic area, defined by the user.

With a variety of businesses needing to customize search results to the preferences of a narrowly targeted audience, there’s an increasing demand for incorporating geographical location information into standard search functionality.

In response, JTeam has recently launched the Solr Spatial Plugin (SSP): a free, standalone, enterprise-ready plugin enabling location based search, built on top of the open source project Apache Solr.

It’s main characteristics are:

• Provide a complete, consistent, robust and fast implementation of advanced geospatial algorithms
• Act as a standalone pluggable extension to Solr
• Written in 100% Java
• Compatible with Apache Solr and Lucene  1
• Open source
• Well documented and comes with support
• Enterprise ready

SSP started out its life as a patch for Solr Spatial Search (Solr-773) and Spatial Lucene (Lucene-1732) and extends Solr and Lucene with previously missing geodetic search functions while improving on the speed of the result and performance when dealing with a large data set.  It does this through better query parsing and multi-threaded filtering; also included are improved extensibility and documentation.

The author of the SSP plugin- JTeam Search specialist Chris Male – explains the technical underpinnings in his earlier blog on Geo-location Search.

General information on the SSP including the Roadmap and  FAQ is  available on the JTeam website.

While the plugin is freely downloadable, JTeam provides installation and configuration support on the SSP as well as annual support contracts for both SSP and Solr.

JTeam can also create customized versions of the SSP, tailored to your needs.

Download the SSP

Contact JTeam for SSP Support and Customization options

1 SSP is currently compatible with Solr 1.4 & Lucene 2.9