\documentclass{elsart} \usepackage{natbib} \usepackage{amsmath} \usepackage{amsfonts} \usepackage{longtable} \bibliographystyle{elsart-harv} \usepackage{url} \def\UrlFont{\rm} \def\UrlLeft#1{#1% \if -\noexpand#1\else \penalty\UrlBreakPenalty \fi \UrlLeft } \def\UrlRight #1\UrlLeft{} % local definitions \newcommand{\best}{_{\rm b}} \newcommand{\worst}{_{\rm w}} \newcommand{\last}{_{\rm l}} \newcommand{\OutS}{\mathbb{O}} \newcommand{\Expe}{{\rm E}} \begin{document} \begin{frontmatter} \title{Information retrieval performance measures for a current awareness report composition aid} \thanks{I am grateful for comments by Christian Calm\`es, William S. Cooper, Robert M. Losee, Amanda Z. Xu and two referees of ``Information Processing and Management''. I am also grateful to all the volunteers of NEP for the work they have been putting into running the service. The hospitality of Tatyana I. Yakovleva provided a congenial setting for the work on this paper.} \author{Thomas Krichel} \address{College of Information and Computer Science\\ Long Island University \\ 720 Northern Boulevard \\ Brookville NY 11548--1300, U.S.A.} \address{Faculty of Information Technology\\ Novosibirsk State University\\ 2, Pirogova Street\\ 630090 Novosibirsk, Russia } \address{http://openlib.org/home/krichel \\ krichel@openlib.org} \begin{abstract} This papers studies a special ``small'' information retrieval problem where user satisfaction only depends on the ordering of documents. We look for a retrieval performance measure applicable for this setting. We define some requirements for such a measure. We develop a theoretical ordering of all outcomes. We look at some standard and purpose-build measures and assess them against the requirements. We conclude that a linear combination of two such measures is adequate. \end{abstract} \end{frontmatter} \section{Introduction} The classic measures of information retrieval performance are precision and recall. Precision is the number of retrieved and relevant documents divided by the number of retrieved documents. Recall is the number of retrieved and relevant documents divided by the number of relevant documents. Both ratios are used jointly because they capture two complementary aspects of the retrieval process. Precision tells us how good the system is at filtering out non-relevant documents. Recall tells us how good the system is at finding relevant documents. Other measures have been proposed that aim to summarize information retrieval performance in a single number. These include the average precision at seen document, the R-precision, the E-measure, van Rijsbergen's F and the average precision over all documents. All of these numbers are directly based on the concepts of precision and recall. In fact, they are precise mathematical functions of precision and recall ratios. Despite numerous critiques of these measures, they remain the most widely deployed in ``large'' information retrieval problems. In such ``large'' problems there is a large set of documents, typically so large that the one-by-one examination of each document is not realistic. Then, an information retrieval system has the task of retrieving a set of documents that corresponds to the information need. The user only sees the set of retrieved documents. This paper is motivated by my concern for a special information retrieval problem. We can call this problem a ``small'' information retrieval problem. The service helps a user to find the relevant documents out of a small collection of documents. The collection is small enough that the user can examine each document one by one. The purpose of the information retrieval system is to make it easier for users to reach decisions. One example of such a ``small'' information retrieval system has been the motivation for this paper. I created the ``NEP: New Economics Papers'' service at \url{http://nep.repec.org} and I am involved in running it. NEP is a current awareness service of the RePEc digital library, see \url{http://repec.org}. It filters new additions to RePEc into weekly subject-specific reports. Each report is edited by a volunteer editor. Each week editors are given a list of new additions to RePEc. From that list, they select the documents that are relevant to the subject of the report. These ``relevant'' documents form an issue of the report. Report issues are circulated via email. The NEP service has been running since 1998. During that time RePEc has grown. So has the list of new additions that appear each week. The median number of new documents per week, over the entire life of the service, is 300. But in recent months, bumper crops of over 600 new documents are not uncommon. With that sort of numbers, we can not require volunteer editors to ponder over each single one for much more than a few seconds. At present, most editors, when composing the report issue, look first at document titles. If a title looks appealing, they may look at the abstract. But sometimes a non-appealing title may hide a relevant document. This would become clear if the editor has read the abstract. However, with large new additions lists it is not realistic to expect the editor to read the entire set of abstracts. A pre-selection by the title is inevitable. It inevitably leads to editorial mistakes. %% %% start of change requested by R1, fix #1 %% % To make life easier for the editors, it would be useful to sort the % list of new additions such as to show editors up-front those documents % that are most likely to be included. To make life easier for the editors, it would be useful to sort the list of new additions in order to show editors up-front those documents that are most likely to be included. %% %% start of change requested %% This system can bring two benefits to editors. First they do not have to labor through the whole of the new additions list. If the algorithm works well, they can skip the tail end. Second, they can increase attention to the documents that the computer has put to the top of the list. This has the potential to eliminate oversight of documents with imaginative titles. To evaluate such a sorting system, we need some measures. Within the specific context, precision and recall do not appear to be useful. There are three basic ways in which one can interpret precision and recall within the NEP context. One approach is to say that they remain constant. Recall is always 100\%, and precision is always equal to the number of documents that are relevant to the subject report, divided by the size of the new additions list. If we take that view, then precision and recall do not depend on the sorting process. %% %% change requested by R1, fix #2 %% % Another approach looks at precision and recall at the level of the % last included document. Another approach looks at precision and recall at the level of the last relevant document. %% %% end of change requested by R1 %% Thus, if the information retrieval system sorts all the relevant documents to the front, then precision and recall are 100\%. If there are some non-relevant documents that are found before the last document that is relevant, precision is the number of relevant documents divided by the position of the last relevant document. But recall is still 100\%, therefore it is not useful. The measure of precision that we reach with this view is isomorph to the Nosel measure that I discuss in Subsection \ref{sec:nosel}. A third approach is to think of the output of the information retrieval system as sets of documents. We distinguish the documents that the information retrieval system has predicted as relevant versus those that it has predicted as non-relevant. We can then calculate precision and recall figures as intended by comparing the sets returned by the information retrieval system with the sets assembled by the editor. The latter are assumed to be correct. There are two problems with this approach. First, all computer-based information retrieval systems rank documents by the likelihood that they are relevant. It is in the very nature of the type of calculations done that such a ranking is produced. %% %% start of change requested by R1, fix #3 %% % Thus, just looking at sets of documents implies that one % deliberately ignores information which is available. % Therefore, if we conceive the information retrieval system output as a couple of sets, we exclude additional information that the system has produced. %% %% end of change at the request of R1 %% This is inconsistent with an assumption of rational behavior of users. Second, it should matter a lot in what order the documents appear in. This second problem is best illustrated by an example. For the sake of illustration, let numbers denote documents that, according to the editor, are relevant, and letters denote documents that are not relevant. Assume that the information retrieval system finds that documents 1 and 2 are relevant. It will sort those to the front. Then $[1,2,3,4,\text{a},\text{b},\text{c},\text{d}]$ and $[1,2,\text{a},\text{b},\text{c},\text{d},3,4]$ %% %% start of change requested by R1, fix #4 %% % have the have the have the %%. %% end of change at the request of R1 %% same precision (100\%) and recall figures (50\%). But the former is perfect for the editor, while the latter is basically useless. Half of the relevant documents are at the front, the other half is at the rear. The editor has to still work through the entire list of new additions to find the all relevant documents. We hope to have convinced the reader of the need for alternative measures. %% %% start of change requested by R2, fix #30 %% The remainder of the analysis therefore does not start with the analysis of precision and recall values, as \citet{keirij74foundation} or \citet{wilsha86on} have done. Instead we use an informal utility maximization approach, as, for example, \citet{wilcoo73on}. In addition, we will consider rationality arguments, an approach that is more often found in economics rather than in information science. %% %% end of change at the request of R2 %% Before we review criteria, we devote two sections to further thinking about the problem. In Section \ref{sec:general} we set out the general framework. In Section \ref{sec:natural} we look more closely at the problem that editors face. We try to establish what order a rational editor may place on the outcomes. In Section \ref{sec:measures} we present alternative measures. In Section \ref{sec:test} we test these measures on the NEP report data using support vector machines. In Section \ref{sec:conclusions} we offer conclusions. \section{The problem}\label{sec:general} % % There are basically two types of evaluation exercises one can do. We % % can call them individual and collective exercises An individual % % exercise associates a value with the individual outcome of a single % % sorting trials. A collective exercise associates a value to a method, % % given that the same method has been used on a large number of sorting % % trials. To calculate the precise of the evaluation for such a method, % % a large number of sorting processes will have to be run. In this % % section, as well as is the remainder of this paper, we only cover % % individual exercises. That is, we require that the evaluation that % % we conduct associates a value with an individual outcome. We can % % then use standard statistical aggregates---such as the mean---to % % aggregate individual trials to evaluate a method. % % % % To formally address the evaluation measures, let us fix some notation. Let there be a vector $x$ called the outcome vector or simply the outcome. It has $n$ elements, $r$ of which take the value $1$, i.e.~they represent relevant documents, and $n-r$ take the value $0$, i.e.~they represent non-relevant documents. The ratio $r/n$ is the generality of the information need. Let us call $\OutS(r,n)$ the outcome set. This is the set of all possible outcomes. For a fixed $r$ and a fixed $n$, this is a set with $n!/(r!\,(n-r)!)$ members. Loosely speaking, we are looking for a measure of how much the 1s are at the front of the vector and the 0s at the end. The best outcome is $$ x\best=[1,\ldots,1,0,\ldots,0] $$ and the worst outcome is $$ x\worst=[0,\ldots,0,1,\ldots,1]. $$ Let $f(x)$ denote a measure of the quality of the outcome. There are many measures that one may define. Each of these measures is subjective in a way. It captures a desired property of the outcome. We can find four requirements that hopefully most people can agree with. Requirement 0 is the most subjective. It says that $f(x)$ must not be too complicated. It should be fairly easy to explain to people what the measure is. Requirement 1 is in some ways a corollary of requirement 0. %% %% start of change requested by R1, fix #5 %% % People are are used to reason that a better outcome mean a % higher $f(\cdot)$. People are used to reason that a better outcome means a higher $f(\cdot)$. %% %% end of change requested by R1 %% Thus $$ f(x)>f(x')\quad\Longleftrightarrow\quad x\mbox{ is better than }x'. $$ Requirement 2 again is a corollary of requirement 0. People are used to reasoning in term of percentages. Therefore it seems adequate to require \begin{equation}\label{eq:best} f(x\best)=1. \end{equation} Apart from the best outcome another benchmark is important. That is the case where $x$ is picked randomly out of $\OutS(r,n)$. %% %% change requested by R1, fix #6 %% % In that case, the measure should be 0 in order for it to convey the % idea that the information retrieval system that achieves a positive % value of $f(\cdot)$ outperforms a random allocation. In that case, the outcomes are of no use. %% %% end of change requested by R1 %% This leads to Requirement 3 \begin{equation}\label{eq:exp} \Expe\,f(x)=0, \end{equation} where $\Expe$ stands for the expected value operator. One technical constraint that comes out of this requirement is that there has to be a closed form for the expected value. Of course, the expected value of any measure $f(x)$ over the finite set $\OutS(r,n)$ can be calculated by computer as the average over all potential outcomes. However, for large $n$ and $r$, the number of members of $\OutS(r,n)$ becomes too large for this to be practical. Current computer technology is simply not powerful enough to accomplish the calculations in reasonable time. Finally, we have an additional desired feature. We refer to this as the scaling property. With a constant generality $r/n$, if $r$ and $n$ are both multiplied by a scale $t\in\Nset$, with $t>1$, we would like to get the same values of $f(\cdot)$ if we construct scaled outcome by repeating each element in the original vector $t$ times. For an example, assume $n=3$, $r=1$ and $x=[0,1,0]$. If $t=2$ we can construct $x'=[0,0,1,1,0,0]$, by simply repeating each element in $x$ $t$ times. It is obvious that we can always make such a scaling transformation and that this transformation is unique. Let $t\otimes x$ denote the scaled outcome vector. If we make such a transformation, it appears natural to wish that $f(x)$ does not change, i.e. $$ \forall\,x\in\OutS(r,n), \quad\forall\,t\in\Nset:\qquad f(x)=f(t\otimes x). $$ If $f(\cdot)$ satisfies to that property, we will say that it scales. \section{Subject editor behavior modeling}\label{sec:natural} The proposed system helps editors of a current awareness service such as NEP. The ultimate judge of performance is therefore the subject report editor. To build general criteria, it is useful to build a model of subject editor behavior. While a full mathematical model would be outside the scope of this paper, we hope to establish some general principles using simple deductive reasoning based on a highly simplified view of the editorial process. An editor faces a list of documents. A document is metadata about a paper plus a link to the full text of that paper. An editor may spend a lot or only a little time on the document. This decision on the level of effort per document is difficult to model. Therefore we will not look at it here. In other words, we assume that the examination of a document is a discreet process i.e., the document is examined or it is not examined. Requirement 1 is in some ways a corollary of requirement 0. People are are used to reason that a better outcome mean a higher $f(\cdot)$. %% %% start of change requested by R1 fix #7 %% % After examining a document, the editor knows whether that document % is relevant or or not. After examining a document, the editor knows whether that document is relevant or not. %% %% end of change requested by R1 %% We further assume that the decision to include a document or to exclude it only depends on the contents of that document. It is independent from the contents of other documents. This reasoning assumes away any learning that may take place while the list of documents is examined. As the editor works through the list of documents, she faces two types of costs. First there is the cost $c_1$ of examining the next document. Without loosing much generality we can assume that $c_1$ remains constant over the report issue composition process. Second, there is the cost of missing relevant documents. Let us loosely call this $c_2$, though it is clear that $c_2$ somehow depends on actual number of documents missed. As the editor moves along the list, she faces an optimal stopping problem. If she stops to examine documents, she no longer suffers the penalty $c_1$ from examining all the following documents. But she faces the penalty $c_2$ of missing relevant papers. If $c_1>>c_2$, the editor will not examine any documents at all, and if $c_2>>c_1$ the editor will examine all documents. In less extreme cases, there is a balancing act. This balancing act is complicated because of the uncertainty surrounding $c_2$. To make further progress in our reasoning, we need to simplify the problem. Let us assume that a magical interface could be built that would remove the uncertainty regarding $c_2$. We could imagine a traffic light sign in the editor's interface. It would show green while there is at least one more relevant document, and it would show red if there is no more relevant documents. Such a scenario is unrealistic of course, but for the moment just imagine it could be achieved. Clearly when the traffic light turns red, the editor will stop examining new documents. Now let us in addition assume that the editor is a conscientious person. By this we mean that while the traffic light is green, she will continue to examine new documents, until the traffic light is red. However unrealistic this scenario of the traffic light scenario is, it can teach us one insight. With a traffic light, the editor will, when presented with two outcomes $x$ and $x'$ prefer the one where the last value of $i$ where $x_i$ is $1$, say $i^*$, $i^*=\text{arg} \max_i: x_i=1$ has the lower value. For a conscientious editor examining either $x$ or $x'$, $c_2=0$. However the examination cost $c_1$ will be lower the lower $i^*$ is. This reasoning establishes a weak ordering over all outcomes. Comparing two outcomes, an editor will prefer the one with the last relevant document at an earlier position. The editor will be indifferent between two outcomes that have the last relevant documents at the same position. This reasoning requires under certainty, and that the editor is conscientious. We can hope that it will also hold under some uncertainty, provided that the uncertainty is not too large, and for less then full conscientious editors, provided they are not reckless. Now imagine that the traffic signal can not be completely trusted. It would get it most of the time right but not always. Let us assume, to simplify, that the uncertainty would only hold between the second-to-last and the last document. Assume that the editor would still follow the rule to stop if the traffic light turns to red, simply because the uncertainty is marginally small. %% %% start of change requested by R1, fix #8 %% Let $i^*$ be the position of the last relevant document. Compare two outcomes that are identical, save the fact that positions $i^*-1$ and $i*-2$ are exchanged \begin{align*} x_1&=[\dots,0,1,1,0,\ldots,0]\\ x_2&=[\dots,1,0,1,0,\ldots,0]. \end{align*} In the notation above $\ldots$ represent positions that are identical for both outcomes. The second $1$ is the last, i.e.~it is followed by zeros only. %% %% end of change requested by R1 %% I claim that when the editor compares the two outcomes, she will prefer $x_2$ over $x_1$. The reasoning goes as follows. If a wrong red signal is perceived at position $i^*-1$, the editor looses the last relevant document under $x_2$, but two relevant documents under $x_1$. If a wrong signal is received at any earlier position than $i^*-1$ the loss of the number of relevant documents is the same. Therefore, when presented with two outcomes that have the same position for the last relevant document, the editor will prefer the one with the second-to-last document at the earlier position. This reasoning can be repeated for the third-to-last document etc. We obtain a complete order over the outcomes. Let us call it the natural order. We can conjecture that it is possible to find a class of functional specifications for the loss function, and a class of distribution functions of documents in the included/excluded domain such that, when editors minimize total loss knowing the distribution function, they prefer outcomes in natural order. Developing such a generic class would, however, go beyond the scope of this paper. \section{Various measures}\label{sec:measures} In this section we address different measures for the literature or purpose-built for this paper. %% %% start of change requested by R2, fix #21 %% % For the sake of clear labeling, each % measure that we will be using for testing in Section \ref{sec:test} is % named after a river in Siberia. The river and the measure are % unrelated, of course. %% %% end of change requested by R2 %% \subsection{The Swets/Brookes measure} One interesting measure was proposed by \citet{jonswe63information}. He assumes that ``when a search query is submitted to a retrieval system, the system assigns an index value (call it $z$) to each item in the store.'' This assumption holds true for all computerized text classification and information retrieval systems. Let $z|\text{r}$ and $z|\text{n}$ be the value that the system assigns to $z$ given that it is relevant and non-relevant. \citet{jonswe63information} proposes to evaluate $$ \frac{\Expe(z|\text{r})-\Expe(z|\text{n})}{\sigma^2(z|\text{n})}, $$ where $\sigma^2$ denotes the variance. %% %% start of change requested by R1, fix #9 %% % According to \citet{jonswe63information} proposes to evaluate, as long % as both conditional distributions of $z$ are normal, the measure has % desirable properties. % According to \citet{jonswe63information}, as long as both conditional distributions of $z$ are normal, the measure has desirable properties. %% %% end of change requested by R1 %% But \citet{berbroo68the} suggests that a better measure would be \begin{equation}\label{eq:brooks} \frac{\Expe(z|\text{r})-\Expe(z|\text{n})}{ \sqrt{\sigma^2(z|\text{r})+\sigma^2(z|\text{n})}}. \end{equation} This measure truly expresses the discriminating power of the underlying information system. The $z$ data that it uses is much richer than the positional data used by other measures proposed in the following here. On the other hand, its precise values are dependent on the technical characteristics of the information retrieval system. Therefore, its best use is as a technical tool to compare different parameters within the same information retrieval methodology. An important problem is that the measure violates requirement two\footnote{It is not trivial %% %% start of change requested by R1 fix #10 %% % to to to %% %% end of change requested by R1 %% normalize the value such that it is 1 in the best case. One approach is that, faced with an outcome $x$ and its associated $z$ vector, we can construct a perfect outcome that would rearrange $x$ to have all relevant documents at the top, while keeping the $z$ values constant. Unfortunately, while the constructed best outcome certainly has a better value in term of the numerator in the expression \ref{eq:brooks}, we can not be sure about which way the denominator is going. Therefore we can not construct an artificial best outcome in this intuitive way.} The measure has one more problem. It is a measure that is essentially linear. The numerator is linear, and the sign of the expression only depends on the numerator. This causes a problem that we discuss in the next subsection. \subsection{The Aselt measure} \begin{table} \caption{The Aselt measure for $r=2$ and $n=5$} \label{table:aselt} \begin{center} \begin{tabular}{@{}ccrr} $x$&$\alpha(x)$&$a(x)$\\ 1,1,0,0,0&1.5&$1.0$\\ 1,0,1,0,0&2.0&$2/3$\\ 0,1,1,0,0&2.5&$1/3$\\ 1,0,0,1,0&2.5&$1/3$\\ 0,1,0,1,0&3.0&$ 0$\\ 0,0,1,1,0&3.5&$-1/3$\\ 1,0,0,0,1&3.0&$ 0$\\ 0,1,0,0,1&3.5&$-1/3$\\ 0,0,1,0,1&4.0&$-2/3$\\ 0,0,0,1,1&4.5&$-1.0$\\ \end{tabular} \end{center} \end{table} Although Swets/Brookes measure is rarely used at present, linear measures have not disappeared. For example, \citet{roblos98text} considers the ``average search length'' as a candidate measure for information retrieval performance. The average search length is the average position of units within the vector $x$. That is, it is the sum of the positions $i$ where $x_i=1$, divided by $n$. If $a(x)$ is the average search length, we have $$ \alpha(x\best)=\frac{1+r}{2}\qquad\text{and} \qquad\alpha(x\worst)=\frac{2\,n-r+1}{2}. $$ The expected value for the average search length\footnote{This could be labelled the average average search length or expected average search length. Our use of %% %% start of change requested by R2, fix #26 %% % geographical names acronyms %% %% start of change requested by R2 %% avoids confusion.} can be readily found as $$ \Expe\,\alpha(x)=\frac{n+1}{2}, $$ which, interestingly, does not depend on $r$. Having found the expected value, we can construct a measure that satisfies requirements 2 and 3. %% %% start of change requested by R2, fix #23 %% % We define the Aselt measure of an outcome $x$, $a(x)$, by We define the Aselt---an acronym for ``\underline{a}verage \underline{se}arch \underline{l}ength \underline{t}ransformed''---measure of an outcome $x$, $a(x)$, by %% %% end of change requested by R2 %% $$ a(x)=\frac{n+1-2\,\alpha(x)}{n-r}. $$ The Aselt measure is nicely bounded $a(x\best)=1$, $a(x\worst)=-1$. It satisfies requirements 1--3. And it scales, because it is based on an average, which itself is a linear function. Table \ref{table:aselt} carries a numeric illustration of the Aselt measure. Looking at it we have two remarks. First, there seems to be a concentration of points towards the middle of the distribution. Second, not every different outcome has a different Aselt measure. In particular, the measure does not penalize two relevant documents in the middle of the vector any different than one relevant document in the end. From a managerial point of view, this is a problem if we are much concerned about having a measure that penalizes heavily a single document that is left out at the end. In Table \ref{table:aselt} outcomes are sorted by the natural order. The table illustrates that the Aselt measure violates the natural order. That is, there are some couples of outcomes $x$ and $x'$, where $x$ is better than $x'$ according to the natural order but $a(x)1$ will be able to accomplish the purpose of implementing the natural order. This motivates the following definition. \begin{table} \caption{Ponori measure for $r=2$ and $n=5$, $y=2$ and $y=\infty$} \begin{center} \begin{tabular}{@{}ccrr}\label{table:ponori} $x$&$\omega(x,2)$&$o(x,2)$&$o(x,\infty)$\\ 1,1,0,0,0&3&$ 1.0$&$1$\\ 1,0,1,0,0&5&$ 37/47$&$1$\\ 0,1,1,0,0&6&$ 32/47$&$1$\\ 1,0,0,1,0&9&$ 17/47$&$1$\\ 0,1,0,1,0&10&$ 12/47$&$1$\\ 0,0,1,1,0&12&$ 2/47$&$1$\\ 1,0,0,0,1&17&$-23/47$&$-1.5$\\ 0,1,0,0,1&18&$-28/47$&$-1.5$\\ 0,0,1,0,1&20&$-38/47$&$-1.5$\\ 0,0,0,1,1&24&$-58/47$&$-1.5$\\ \end{tabular} \end{center} \end{table} Let $x=[x_1,\ldots,x_n]\in\OutS(r,n)$. %% %% start of change requested by R2, fix #24 %% % Then the $y$ Ponori penalty of $x$, $\omega(x,y)$ is Then the $y$-Ponori---an acronym for ``\underline{po}lynomial \underline{n}atural \underline{or}der \underline{i}mposition''---penalty of $x$, $\omega(x,y)$ is $$ \omega(x,y)=\sum_{i=1}^{n} y^{i-1}\,x_{i}. $$ We find \begin{equation}\label{eq:ponoriest} \omega(x\best,y)=\frac{y^{r}-1}{y-1} \qquad\text{and}\qquad \omega(x\worst,y)=y^{n-r}\,\omega(x\best,y). \end{equation} The expected value is \begin{equation}\label{eq:oexp} \Expe\,\omega(x)=\frac{y^{n}-1}{y-1}\,\frac{r}{n}. \end{equation} %% %% start of change requested by R1, fix #12 %% We can substitute for (\ref{eq:ponoriest}) and (\ref{eq:oexp}) to obtain a measure that satisfies (\ref{eq:best}) and (\ref{eq:exp}). %% %% end of change requested by R1 %% This motivates the following definition. The Ponori measure of an outcome $x$ at the power $y$ is $$ o(x,y)=\frac{(y^n-1)\,r-(y-1)\,n\,\omega(x,y)}{(y^n-1)\,r-n\,(y^r-1)}. $$ As $y\to\infty$ the Ponori measure becomes as indicator if the last relevant document is at the last position. As $y\to1$ $o(x,y)\to a(x)$. Thus, the Aselt measure is nothing but a limiting case of the Ponori measure. In this limiting case, the Ponori measure scales. But with $y>1$ it does not scale. \subsection{The Copnori measure} The dependency of the Ponori measure on $y$ is inconvenient. It is not clear what $y$ to choose. While the ordering of outcomes is not sensitive to the choice of any $y>1$, the numbers coming out of the evaluation definitely are. Thus a more fundamental measure is called for. One very simple way to achieve this is to count through the elements in the outcome set in the natural order, assigning each worse outcome an incremental penalty of 1. Computing the sequence is reasonably straightforward\footnote{The detail of our computational implementation is as recursive. For any outcome vector, we first remove the trailing non-relevant outcomes. They will not affect the result. Thus we have a shortened outcome vector with $n_1$ elements, say, $r$ of which are relevant. We can then calculate a minimum value for $\kappa(x)$ as $\kappa\best(1)=(n_1-1)!/(n_1-1-r)/r!$. We remove the last relevant outcome from the vector. This completes the first step. We have a new vector of $n_1-1$ element, $r-1$ of which are relevant. Again, we remove any non-relevant outcomes of the end of that new vector. We find the next relevant outcome at $n_2$. We have a new minimum value, $\kappa\best(2)$ which we add to the value found in the previous step $\kappa\best(1)$, etc. We continue proceeding until we arrive at a vector that has only relevant outcomes. There we find the sum of all $\kappa\best(\iota)$, where $\iota$ is the step number.} If we start with counting at 0, we get the counts $\kappa(x)$ as \begin{equation}\label{eq:kbest} \kappa(x\best)=0\qquad\text{and}\qquad \kappa(x\worst)=\frac{n!}{r!\,(n-r)!}-1. \end{equation} \begin{table}\label{table:nisa} \caption{The Copnori measure for $r=2$ and $n=5$} \begin{center} \begin{tabular}{@{}ccrr} $x$&$\kappa(x)$&$k(x)$\\ 1,1,0,0,0&0&$1.0$\\ 1,0,1,0,0&1&$7/9$\\ 0,1,1,0,0&2&$5/9$\\ 1,0,0,1,0&3&$1/3$\\ 0,1,0,1,0&4&$1/9$\\ 0,0,1,1,0&5&$-1/9$\\ 1,0,0,0,1&6&$-1/3$\\ 0,1,0,0,1&7&$-5/9$\\ 0,0,1,0,1&8&$-7/9$\\ 0,0,0,1,1&9&$-1.0$\\ \end{tabular} \end{center} \end{table} %% %% start of change requested by R2, fix #27 %% % To find the expected value is really easy The expected value is readily found as %% %% start of change requested by R2, fix #27 %% \begin{equation}\label{eq:kexp} \Expe\,\kappa(x)=\frac{\kappa\worst+\kappa\best}{2}. \end{equation} %% %% start of change requested by R1 fix #13 %% We can substitute for (\ref{eq:kbest}) and (\ref{eq:kexp}) to obtain a measure that satisfies (\ref{eq:best}) and (\ref{eq:exp}). %% %% end of change requested by R1 %% This motivates a definition. %% %% start of change requested by R2 fix #25 %% % The Copnori measure $k(x)$ of an outcome $x\in\OutS(r,n)$ is The Copnori---an acronym for ``\underline{co}nstant \underline{p}enalty \underline{n}atural \underline{or}der \underline{i}mposition''---measure $k(x)$ of an outcome $x\in\OutS(r,n)$ is %% %% end of change requested by R2 %% $$ k(x)=1-\frac{2\,(\kappa(x))}{\frac{n!}{r!\,(n-r)!}-1}. $$ Unfortunately, the Copnori measure does not scale. %% %% start of change requested by R2, fix #28 %% % This is easily seen with an example. This is seen with an example. %% %% end of change requested by R2 %% $$ k(0,1,0,1,0)=1/9, \qquad k(0,0,1,1,0,0,1,1,0,0)=89/209. $$ %% %% start of change requested by R2, fix #29 %% % which is not even close. Both numbers are not even close to each other. %% %% end of change requested by R2 %% But, the measure has three strong points. First, if one understands the natural order, it is a very intuitive measure. Second, it does not depend on an arbitrary parameter. Third, given the algorithm that we developed, the Copnori measure is easy to compute even for large $n$ and $r$. \section{Test}\label{sec:test} Our aim is to develop a sorted list of new additions to RePEc for editors of NEP. In NEP each subject issue has a code nep-{\em xxx} where {\em xxx} is a sequence of three letters. A special report nep-all contains the list of all new additions to RePEc. Thus sorting the list of new addition is like sorting the nep-all report issue. Each time a new nep-all issue is produced, it is sorted for the use of the editors. For each subject report, the result of the sorting is different, of course\footnote{In NEP documentation the term ``pre-sorting'' rather than ``sorting'' is used. In NEP ``sorting'' is a different process than pre-sorting. Sorting occurs when an subject issue is being produced. After an editor has discarded non-relevant documents, (s)he may decide to sort the documents in an order such as to put the most interesting document right to the top of the issue. This is an optional step of the process of creating a new subject issue.}. To sort the nep-all for a subject report we look at the past subject report issue data. For each subject report, we have two sets. The first is the set of documents that have been included in the subject report. The second is the set of documents that have not been included in the report. The membership of the latter set is somewhat more difficult to determine than the former. Sometimes, an editor may not have looked at an entire list of new documents. This can happen, for example, if the editorship of a report is vacant. Therefore we restrict membership of the second set to all those documents in the nep-all issues for which at least one document of the nep-all issue has been included from. Thus documents in nep-all issues in which no document appeared in the subject issue have been ignored. While this may be an oversight of negative learning examples, there are still plenty of negative example left, because the generality of subject reports is small. We treat the occurrence of documents in different reports as independent events. Data in \citet{barrueco03organizing} suggests that this is not the case. However, in a practical application, it would be cumbersome to rerank a nep-all report for a certain subject when it becomes known that the editor of another subject report has included that document in her report issue, based on that new information, because editors make their decisions independently from each other but typically within a short time frame after the nep-all report has been issued. Thus, by ignoring co-occurrence of documents altogether, we are working under realistic operating conditions. We keep feature extraction very simple. From each document, we use the author names, title, abstract, classification codes, and the serial in which the paper has appeared. We concatenate the resulting string. We remove all punctuation, transliterate to lowercase and collapse whitespace. Each whitespace-separated component of the resulting string is a feature. %%% %%% start of change requested by R1, fix #15 %%% % We weigh feature by the straight frequency counts first. Then we norm % the features such that the Euclidean norm of the feature vector is % unit. To form the ranking, For each document, we count the occurrence of the feature $f$ as $t_f$. The weight of the feature $f$ in the document, $w_f$ is then given as $$ w_f=\frac{t_f}{\sum_{\forall f} t_f^2}. $$ Note that it is not necessary to take document frequency into account here, because to form the ranking, %%% %%% end of change requested by R1 %%% we use support vector machines (SVM). This technique goes back to \citet{servap95nature}. It is now a widely used text classification technique. \citet{paugin04mapping} provide one example in a similar context to ours. The svm\_light software of \citet{torjoa99making} runs all the calculations. According to \citet{tomkri05developing} the median nep-all report has 300 documents. %%% %%% start of change requested by R1, fix #16 %%% % Thus each run, we use 300 documents for testing, % the rest for training. We conduct 100 runs. Therefore we set aside 300 randomly selected documents for testing. The rest we use for training the SVM. We conduct at least 10 runs for each report. For some reports, where the generality is low, some selected testing dataset contains no relevant document. In that case, we repeat runs until at least one among the 300 randomly selected testing documents is relevant. %%% %%% end of change requested by R1, fix #17 %%% The results are so bulky that we have confined them to an appendix \ref{sec:results}. %% %% start of change requested by R1 fix #18 %% % While the Ponori measure has desirable theoretical properties, there is % a bad problem in tests when the set of outcomes to be ordered is large, % say more that 100. The Aselt measure gives a reasonable range of results, and shows, by its numerical values, that the performance of the SVM is really quite good. But we have rejected the measure on theoretical grounds. The same holds for the Lofop measure. We still include them in Table \ref{table:result} for the sake of completeness. While the Ponori measure has desirable theoretical properties, there is a bad problem in tests where the set of outcomes to be ordered is large, say more than 100. %% %% end of change requested by R1 %% In that case, if $y>1$, any practical outcome that has the ability to lift the last document to say before the last third or last quarter of all the documents will get a measure that is close to 1, or even equal to 1 after rounding. As we increase the value of $y$, we are converging toward a situation where the outcome is $100.00\%$ as soon as the last document is not relevant, and a negative number if it is. This clearly is not what we want. A similar problem affects the Copnori measure. Recall that the Copnori measure gives each outcome its own entire number. Outcomes where a relevant document appears late are penalized very heavily. %% %% start of change requested by R1 #14 %% % Therefore, as soon as the retrieval system is able lift all relevant % documents from low positions, it does very well. Even if the last % relevant document is is the middle, the measure shows a result that is % close to 100%. Therefore, as soon as the retrieval system is to able lift up all relevant documents from low positions, it does very well. Even if the last relevant document is in the middle, the measure shows a result that is close to 100\%. %% %% end of change requested by R1 %% In fact, is does so even more than the Ponori measure. The alert reader will note that there are a number of maxima in the table are 100\% for the Copnori measure but less than 100\% for the others. In the theoretical framework that we use, 100\% is the value reserved for the optimal outcome $x\best$. It is therefore impossible for the same outcome to be evaluated 100\% by one measure and less than 100\% by another measure. The explanation for this apparent error is the table is rounding. The computer says its 100\% when in fact is can not see any more the real value that is a tiny bit below 100\%. The Nosel measure somehow has the opposite problem. It only looks at the last position of the last document, it takes no account of the ability of the information retrieval system to put relevant documents to the front. However as we noted in the introduction, the feature is also important, because it allows the editor to spend extra efforts on the documents, reading more than the title. Thus we can say that in principle, the Nosel measure ``underestimates'' the success of the system, whereas the Copnori measure ``overestimates'' the success of the system. However, this general statement only holds when the system is a success. When the result is lousy, Copnori exaggerates the bad performance. This is simple the reverse of the fact that the Copnori measure gives very good results for a large span of top-measures. Since both Copnori and Nosel have an expected value of zero at the random results, the Copnori compensates with more lower values at the tail end. Related to this, the variance of the Copnori measure is higher than the variance of other measures. Generally, when the results are quite good, the Nosel measure has the higher variance. This comes as no surprise since it only looks at one single element of the outcome vector, the one that comes lowest. \section{Conclusions}\label{sec:conclusions} We think of precision and recall as set-based measures. Indeed, they are based on the idea that the total set of documents contains two complementary subsets, the subset of relevant documents and the subset of non-relevant documents. A query creates two other complementary subsets, the subset of retrieved and the subset of non-retrieved documents. In this paper, we discuss a different class of information retrieval performance measures we call vector-based measures. Vector-based measures start with a different way of thinking about what is happening at query time. We think of the set of documents as a vector. Indeed, from a computational point of view, it is a vector, because all documents are in some order in the information retrieval system. The task of the information retrieval system is to sort all the documents that are relevant to the beginning of the vector, and sort the non-relevant documents to the end of the vector. From our setup we have a theoretical ordering of outcomes we call the natural order. Therefore to evaluate the information system, we prefer measures that respect the natural order. %% %% start of change requested by R1 fix #19 %% % The Nosel measure only weakly enforces the natural order. It % undervalues the performance of the system. The Copnori measure strictly % imposes the natural order. But since it attaches a constant penalty % with each step, and therefore does not properly take account of the % extra effort to examine the next document. Thus it appears best to % take a linear combination of the two measures $\nu\,l+(1-\nu)\,k$. As % long as $0<\nu\le1$, the measure strictly respects the natural % order. We suggest $\nu=10\%$, but other values are just as % acceptable. All they change is the numeric value of the measure. They % have no impact on the actual ordering of outcomes. % It turns out that the Nosel and Copnori measure complement each other to provide a reasonable approximation of what the editors should want. The Nosel measure only weakly enforces the natural order. A very large number of outcomes receive an identical Nosel measure despite the fact that the arrive at different positions in the natural order. From the point of view of the editors, the Nosel measure only penalizes an outcome when the editor has to look at an additional document. It does not take into account, that, for a given value of the position of the last relevant document, the editor may abandon the search for relevant documents before the last relevant document is reached, and may, as a consequence of this action, have a varying number of documents lost in different outcomes that receive the same Nosel penalty. Such differentiation is provided by the Copnori measure. It strictly enforces the natural order. Each outcomes is assigned a different number and the next worse outcome has a constant additional penalty of one. Moving the last position one further imposes no special additional penalty. But from the point of view of the editors, examining a new document does carry something additional to just one step down in the natural order. From their point of view it has to be a special penalty. Such a extra penalty is provided by the Nosel measure. %% %% end of change requested by R1 %% Therefore it appears best to take a linear combination of the two measures, such as say $\nu\,l(x)+(1-\nu)\,k(x)$. As long as $0<\nu\le1$, the measure strictly respects the natural order, and gives an extra penalty at each extra document that has to examined in order to find all the relevant documents. We suggest $\nu=10\%$, but other values are just as acceptable. All they change is the numeric value of the measure. They have no impact on the actual ordering of outcomes. \appendix \section{Test results}\label{sec:results} In this table, we report, for a selection of NEP reports, the summary statistics for each measure. $a$ is the Aselt measure, $m$ the Lofop measure, $l$ the Nosel measure, $k$ the Copnori measure, and $o$ the Ponori measure. Reports are ordered by generality. To reduce the size of the table, we omitted three out of four subject reports. For each report and each measure, we see the mean in the line ``mean'', the minimum in the line ``min'', the maximum in the line ``max'', and the standard deviation in the line ``dev''. {\small \begin{center} \setlength{\parskip}{0pt} \begin{longtable}{@{}rrrrrrr}\label{table:result} report\phantom{ mean}&$a$&$m$&$l$&$k$&$o(1.01)$\\ {\tt nep-mac} mean&80.21&84.11&21.73&93.19&84.74\\ min&67.38&65.58&\phantom{0}0.85&39.17&67.29\\ max&90.24&93.66&50.55&99.99&94.41\\ dev&\phantom{0}7.69&\phantom{0}8.81&17.43&19.00&\phantom{0}9.07\\ %{\tt nep-fin} mean&77.26&82.95&23.93&96.45&83.83\\ % min&65.23&76.13&\phantom{0}4.04&76.30&76.00\\ % max&89.94&94.34&66.04&99.99&95.84\\ % dev&\phantom{0}7.09&\phantom{0}6.18&18.68&\phantom{0}7.23&\phantom{0}6.51\\ %{\tt nep-bec} mean&55.87&65.48&25.19&77.41&68.58\\ % min&33.68&17.31&$-$7.95&$-$84.68&29.27\\ % max&72.40&79.67&49.04&99.97&81.51\\ % dev&12.54&18.66&17.03&57.25&16.25\\ %{\tt nep-pbe} mean&82.19&88.75&51.53&99.60&90.63\\ % min&71.29&79.29&17.31&96.15&80.46\\ % max&89.49&94.23&80.37&99.99&96.13\\ % dev&\phantom{0}6.83&\phantom{0}5.06&18.74&\phantom{0}1.21&\phantom{0}5.02\\ {\tt nep-lab} mean&81.08&86.16&34.23&98.45&86.96\\ min&73.16&73.07&\phantom{0}6.96&86.68&71.20\\ max&88.88&93.89&79.31&99.99&95.79\\ dev&\phantom{0}5.70&\phantom{0}6.40&20.98&\phantom{0}4.16&\phantom{0}7.49\\ %{\tt nep-geo} mean&92.48&95.01&71.12&97.53&95.88\\ % min&82.89&83.29&\phantom{0}6.04&75.35&82.90\\ % max&98.67&99.31&95.88&100.00&99.57\\ % dev&\phantom{0}5.09&\phantom{0}4.72&26.78&\phantom{0}7.79&\phantom{0}4.92\\ %{\tt nep-sea} mean&88.25&89.46&59.28&78.54&91.45\\ % min&79.49&62.33&$-$6.29&$-$80.46&73.67\\ % max&98.15&99.02&91.36&99.99&99.38\\ % dev&\phantom{0}8.05&11.47&34.78&56.87&\phantom{0}9.19\\ %{\tt nep-dev} mean&81.02&85.30&41.39&87.15&86.49\\ % min&66.10&71.56&$-$3.18&$-$11.08&72.86\\ % max&92.31&95.49&83.91&99.99&97.06\\ % dev&\phantom{0}9.16&\phantom{0}9.13&28.79&34.83&\phantom{0}9.44\\ {\tt nep-ure} mean&89.62&93.59&64.09&99.86&94.79\\ min&82.63&88.15&36.99&98.71&88.97\\ max&97.51&98.72&96.53&99.99&99.25\\ dev&\phantom{0}5.31&\phantom{0}3.70&18.88&\phantom{0}0.40&\phantom{0}3.50\\ %{\tt nep-mon} mean&92.06&94.50&55.50&99.75&94.95\\ % min&85.12&88.48&19.27&98.60&88.39\\ % max&98.65&99.29&95.12&100.00&99.56\\ % dev&\phantom{0}4.39&\phantom{0}3.65&28.07&\phantom{0}0.46&\phantom{0}3.97\\ %{\tt nep-ecm} mean&96.16&97.83&84.30&99.99&98.46\\ % min&92.33&95.51&54.72&99.99&96.21\\ % max&98.68&99.32&97.37&100.00&99.59\\ % dev&\phantom{0}2.18&\phantom{0}1.37&13.84&\phantom{0}5.80e$-$05&\phantom{0}1.15\\ {\tt nep-eec} mean&74.72&79.83&26.12&83.03&80.68\\ min&55.68&57.14&$-$1.38&\phantom{0}8.50&57.12\\ max&84.59&90.94&60.13&99.99&93.03\\ dev&\phantom{0}9.94&11.30&21.65&30.53&12.08\\ %{\tt nep-mic} mean&61.71&72.02&25.41&96.75&73.35\\ % min&45.92&55.02&10&87.55&53.66\\ % max&82.23&89.54&54.33&99.99&92.00\\ % dev&10.04&\phantom{0}9.68&16.51&\phantom{0}4.43&11.08\\ %{\tt nep-com} mean&78.69&84.79&42.86&98.58&85.66\\ % min&64.68&69.72&14.29&91.85&67.53\\ % max&96.44&98.04&80.89&99.99&98.65\\ % dev&10.99&\phantom{0}8.88&24.23&\phantom{0}2.69&\phantom{0}9.45\\ {\tt nep-tra} mean&92.52&93.94&65.77&93.28&94.22\\ min&75.76&73.46&\phantom{0}0.81&34.45&72.68\\ max&99.79&99.89&99.24&100.00&99.93\\ dev&\phantom{0}7.70&\phantom{0}8.12&36.01&20.67&\phantom{0}8.49\\ %{\tt nep-int} mean&95.67&97.25&79.02&99.71&97.69\\ % min&85.65&88.55&28.31&97.10&87.95\\ % max&98.50&99.23&97.96&99.99&99.56\\ % dev&\phantom{0}3.73&\phantom{0}3.15&20.30&\phantom{0}0.91&\phantom{0}3.48\\ {\tt nep-ino} mean&88.04&91.31&57.95&91.44&92.40\\ min&66.90&66.70&$-$1.16&14.74&67.77\\ max&98.34&99.12&93.36&99.99&99.46\\ dev&\phantom{0}9.35&\phantom{0}9.47&30.28&26.95&\phantom{0}9.37\\ %{\tt nep-his} mean&68.34&75.18&22.43&90.64&75.37\\ % min&54.97&61.56&\phantom{0}7.58&75.45&60.00\\ % max&87.34&92.07&49.71&99.99&93.42\\ % dev&\phantom{0}9.86&\phantom{0}9.47&15.96&\phantom{0}8.82&10.59\\ %{\tt nep-env} mean&91.26&94.07&62.27&99.27&94.68\\ % min&80.26&84.78&19.08&95.86&84.87\\ % max&98.73&99.33&96.60&99.99&99.59\\ % dev&\phantom{0}5.38&\phantom{0}4.47&27.05&\phantom{0}1.48&\phantom{0}4.85\\ %{\tt nep-acc} mean&73.97&80.13&35.02&89.41&80.80\\ % min&53.58&61.06&\phantom{0}6.68&61.67&59.28\\ % max&89.16&94.03&80.14&99.99&95.94\\ % dev&11.13&11.30&26.64&13.26&13.06\\ {\tt nep-fmk} mean&90.45&94.29&71.04&99.96&95.58\\ min&79.51&87.71&32.19&99.62&90.48\\ max&97.29&98.60&94.71&99.99&99.15\\ dev&\phantom{0}5.93&\phantom{0}3.76&17.39&\phantom{0}0.11&\phantom{0}3.25\\ %{\tt nep-ets} mean&91.82&92.99&75.30&89.40&93.22\\ % min&72.11&64.15&$-$4.14&$-$2.62&61.05\\ % max&99.57&99.78&99.62&100.00&99.87\\ % dev&10.48&11.66&34.54&32.35&12.66\\ %{\tt nep-pol} mean&82.23&86.14&52.24&83.44&87.00\\ % min&68.25&64.67&$-$3.47&$-$3.12&64.43\\ % max&96.88&98.40&96.74&99.99&99.07\\ % dev&10.78&11.12&35.18&32.35&11.97\\ %{\tt nep-cfn} mean&81.74&86.39&48.50&89.30&87.12\\ % min&55.02&59.81&\phantom{0}3.78&45.72&58.11\\ % max&94.26&96.98&90.36&99.99&98.14\\ % dev&12.03&11.95&28.69&20.35&13.28\\ {\tt nep-ifn} mean&92.14&94.13&57.22&98.07&94.45\\ min&83.91&86.67&\phantom{0}5.45&81.24&85.69\\ max&97.69&98.78&92.02&100.00&99.21\\ dev&\phantom{0}5.07&\phantom{0}4.79&31.32&\phantom{0}5.91&\phantom{0}5.29\\ %{\tt nep-reg} mean&67.18&76.94&36.85&87.05&79.55\\ % min&28.70&36.79&$-$3.06&$-$6.86&40.02\\ % max&86.96&92.85&74.96&99.99&95.21\\ % dev&15.98&15.50&20.01&33.35&15.31\\ %{\tt nep-hpe} mean&87.95&91.39&68.45&89.24&93.00\\ % min&78.48&69.96&$-$3.55&$-$6.67&69.51\\ % max&96.20&98.00&91.05&99.99&98.78\\ % dev&\phantom{0}6.40&\phantom{0}8.23&28.59&33.70&\phantom{0}8.67\\ %{\tt nep-cmp} mean&85.08&90.75&67.50&98.31&92.49\\ % min&70.16&78.37&28&84.36&79.02\\ % max&96.09&97.96&92.97&99.99&98.77\\ % dev&10.01&\phantom{0}6.98&23.32&\phantom{0}4.91&\phantom{0}6.64\\ %{\tt nep-ene} mean&89.30&93.41&74.81&99.50&94.71\\ % min&69.61&80.67&36.42&97.82&83.84\\ % max&99.45&99.72&99.59&99.99&99.84\\ % dev&\phantom{0}9.60&\phantom{0}6.32&22.45&\phantom{0}0.82&\phantom{0}5.57\\ %{\tt nep-edu} mean&85.21&89.84&53.97&95.91&91.07\\ % min&75.67&78.04&\phantom{0}4.93&64.72&78.40\\ % max&97.09&98.50&95.79&99.99&99.10\\ % dev&\phantom{0}7.26&\phantom{0}6.26&28.57&11.04&\phantom{0}6.46\\ {\tt nep-cba} mean&87.59&92.49&60.88&99.92&94.05\\ min&78.92&86.56&40.33&99.50&88.66\\ max&95.28&97.55&93.62&99.99&98.51\\ dev&\phantom{0}5.34&\phantom{0}3.52&17.13&\phantom{0}0.15&\phantom{0}3.18\\ %{\tt nep-dge} mean&89.35&93.53&65.70&99.68&94.89\\ % min&77.60&83.50&25.63&96.89&83.87\\ % max&93.69&96.53&84.97&99.99&97.80\\ % dev&\phantom{0}5.04&\phantom{0}3.96&16.32&\phantom{0}0.97&\phantom{0}4.15\\ %{\tt nep-ent} mean&88.87&93.26&66.22&99.98&94.67\\ % min&78.06&85.78&44.25&99.84&87.74\\ % max&95.79&97.75&90.75&99.99&98.57\\ % dev&\phantom{0}6.32&\phantom{0}4.18&17.10&\phantom{0}0.04&\phantom{0}3.73\\ %{\tt nep-afr} mean&67.01&71.43&39.11&66.47&74.01\\ % min&22.28&21.86&$-$11.72&$-$84.41&30.75\\ % max&97.63&98.79&97.88&99.99&99.31\\ % dev&23.31&25.79&37.97&57.53&23.70\\ %{\tt nep-hea} mean&97.97&98.88&93.36&99.93&99.23\\ % min&92.48&95.38&62.82&99.33&96.31\\ % max&99.51&99.75&99.33&100.00&99.85\\ % dev&\phantom{0}2.17&\phantom{0}1.34&11.38&\phantom{0}0.21&\phantom{0}1.08\\ %{\tt nep-agr} mean&85.80&88.62&60.99&86.04&89.53\\ % min&55.05&49.16&$-$5.56&$-$17.96&47.92\\ % max&95.72&97.76&91.27&99.99&98.62\\ % dev&12.76&14.94&32.22&37.16&15.81\\ {\tt nep-tid} mean&76.93&84.14&47.17&90.77&86.33\\ min&53.25&69.58&$-$1.56&10.17&71.03\\ max&93.39&96.49&87.16&99.99&97.77\\ dev&11.42&\phantom{0}8.95&24.42&28.32&\phantom{0}8.64\\ %{\tt nep-rmg} mean&77.38&83.83&46.47&88.69&85.79\\ % min&59.34&60.90&$-$2.54&$-$6.79&64.97\\ % max&93.54&96.53&81.28&99.99&97.73\\ % dev&11.08&10.97&26.33&33.59&10.93\\ %{\tt nep-cbe} mean&88.97&93.33&76.06&99.70&94.77\\ % min&68.92&78.94&39.15&97.48&80.85\\ % max&96.11&97.98&95.93&99.99&98.79\\ % dev&\phantom{0}8.24&\phantom{0}5.92&21.13&\phantom{0}0.78&\phantom{0}5.77\\ %{\tt nep-law} mean&88.20&90.82&57.39&89.41&91.50\\ % min&74.64&75.24&$-$0.77&17.55&75.74\\ % max&98.11&99.02&95.37&99.99&99.41\\ % dev&\phantom{0}8.17&\phantom{0}8.45&31.92&26.11&\phantom{0}8.99\\ {\tt nep-gth} mean&94.48&96.76&79.96&99.99&97.56\\ min&86.33&91.27&49.47&99.96&92.56\\ max&99.55&99.76&98.12&100.00&99.85\\ dev&\phantom{0}3.99&\phantom{0}2.62&17.33&\phantom{0}0.01&\phantom{0}2.34\\ %{\tt nep-exp} mean&99.66&99.83&99.48&99.99&99.90\\ % min&99.14&99.56&98.73&99.99&99.74\\ % max&100.00&100.00&100.00&100.00&100.00\\ % dev&\phantom{0}0.33&\phantom{0}0.17&\phantom{0}0.38&\phantom{0}3.24e$-$06&\phantom{0}0.09\\ %{\tt nep-ltv} mean&76.20&83.16&45.58&89.39&85.10\\ % min&61.72&69.44&$-$0.93&16.09&71.10\\ % max&92.06&95.80&86.48&99.99&97.41\\ % dev&\phantom{0}9.87&\phantom{0}9.07&31.40&25.99&\phantom{0}9.52\\ %{\tt nep-dcm} mean&96.20&97.76&89.05&99.81&98.33\\ % min&85.13&91.26&55.55&98.77&93.15\\ % max&100.00&100.00&100.00&100.00&100.00\\ % dev&\phantom{0}5.39&\phantom{0}3.31&15.62&\phantom{0}0.41&\phantom{0}2.66\\ {\tt nep-cwa} mean&65.75&68.63&25.00&54.36&70.57\\ min&23.64&27.54&$-$13.11&$-$81.88&25.32\\ max&93.19&96.38&88.88&99.99&97.73\\ dev&18.24&21.93&30.73&61.38&20.82\\ %{\tt nep-net} mean&80.68&87.40&62.42&95.01&89.17\\ % min&50.84&64.45&19.64&70.70&65.74\\ % max&98.75&99.36&98.41&99.99&99.63\\ % dev&14.83&10.94&27.53&10.05&10.86\\ %{\tt nep-eff} mean&88.16&92.58&78.04&96.43&94.06\\ % min&61.84&78.19&12.15&72.38&77.44\\ % max&98.65&99.31&97.75&99.99&99.60\\ % dev&11.79&\phantom{0}7.89&26.49&\phantom{0}8.60&\phantom{0}7.52\\ %{\tt nep-pub} mean&65.02&71.70&34.40&76.83&72.19\\ % min&\phantom{0}9.01&20.01&$-$1.38&\phantom{0}8.72&15.49\\ % max&94.83&97.24&85.95&99.99&98.25\\ % dev&24.89&23.32&33.23&36.36&25.48\\ %{\tt nep-pke} mean&57.95&67.55&25.88&77.64&68.59\\ % min&38.59&50.60&\phantom{0}0.59&26.56&46.34\\ % max&77.27&86.68&67.62&99.99&90.11\\ % dev&11.80&13.47&23.64&28.00&16.24\\ {\tt nep-evo} mean&89.92&93.97&71.19&99.49&95.35\\ min&78.71&85.75&43.05&95.08&87.29\\ max&98.10&99.03&97.15&99.99&99.43\\ dev&\phantom{0}5.94&\phantom{0}4.01&16.45&\phantom{0}1.54&\phantom{0}3.65\\ %{\tt nep-ind} mean&68.11&76.09&34.91&92.02&77.02\\ % min&43.53&47.80&\phantom{0}4.18&59.82&44.62\\ % max&89.96&94.68&84.13&99.99&96.69\\ % dev&14.42&14.36&26.19&13.70&16.15\\ %{\tt nep-lam} mean&60.64&64.96&28.68&58.10&68.17\\ % min&0&0&$-$13.11&$-$81.88&0\\ % max&88.69&93.11&59.54&99.93&94.55\\ % dev&24.62&29.07&27.24&59.12&28.51\\ %{\tt nep-cis} mean&97.08&98.30&92.25&99.50&98.75\\ % min&88.06&92.80&64.20&95.98&94.44\\ % max&100.00&100.00&100.00&100.00&100.00\\ % dev&\phantom{0}4.80&\phantom{0}2.88&14.51&\phantom{0}1.25&\phantom{0}2.22\\ {\tt nep-cdm} mean&69.30&74.01&35.95&80.52&75.88\\ min&20.29&$-$4.61&$-$14.28&$-$74.47&$-$0.08\\ max&95.25&97.47&87.38&99.99&98.41\\ dev&21.32&29.15&26.40&54.54&28.23\\ %{\tt nep-res} mean&89.09&92.42&82.71&91.14&93.12\\ % min&47.02&53.36&\phantom{0}2.13&21.40&48.59\\ % max&97.46&98.69&95.35&99.99&99.23\\ % dev&14.91&13.76&28.40&24.64&15.65\\ %{\tt nep-cul} mean&90.02&93.87&83.55&96.46&95.12\\ % min&65.65&78.45&39.84&81.59&81.78\\ % max&100.00&100.00&100.00&100.00&100.00\\ % dev&14.09&\phantom{0}8.92&23.75&\phantom{0}6.86&\phantom{0}7.48\\ %{\tt nep-ias} mean&78.09&80.20&53.60&68.62&79.55\\ % min&52.80&49.79&$-$7.68&$-$10.92&42.59\\ % max&100.00&100.00&100.00&100.00&100.00\\ % dev&20.07&21.05&46.36&45.36&23.56\\ {\tt nep-mfd} mean&74.58&79.43&32.29&79.67&80.05\\ min&57.55&65.30&$-$4.19&$-$21.39&67.39\\ max&92.74&95.96&82.27&99.99&97.24\\ dev&11.08&10.66&28.94&36.99&11.03\\ %{\tt nep-spo} mean&84.26&84.56&77.21&79.75&84.14\\ % min&0&0&$-$4.72&0&0\\ % max&100.00&100.00&100.00&100.00&100.00\\ % dev&32.52&32.98&42.57&41.43&33.86\\ \end{longtable} \end{center} } \bibliography{sendai} % % , for the reasons that we have outlined above. The position of all % % the documents should matter in the assessment of performance. % % % % For another illustration, consider a response from a web search % % engine. Most people never go to the end of the list of responses from % % a search engine. What matters to them is that the first few results % % are relevant. As far as they are concerned, the idea of a set of % % retrieved that they can examine is pointless. What counts is the % % position of these responses. Responses beyond the 10th results pages % % have little chance to be used. Beyond a very small number of pages, % % the user will not look at any result. Thus, the results set may be % % very large, in fact we can could think about it as an ordered list % % of the pages of the Web. But the ordering is crucial. The % % set-theoretic measures, but discarding the ordering, leave out % % valuable information that can be used to get a different idea of % % information retrieval performance. % % % % There is one common feature between these illustration of the interest % % in vector-based measures. Vector-based measures are useful when recall % % is of little or no importance. In the NEP example recall is not % % important because a conscientious editor can always go through every % % document. In the search engine example, recall is so difficult to % % estimate, and potentially so large, that with the best of all % % intentions, it does not make much sense to examine it any closer. % % % % % % An example may illustrate the conceptual change. Let numbers denote % % relevant documents and letters denote non-relevant documents. In a % % set-based approach, let the information retrieval system retrieve % % ${1,a,2,b,3}$. Let it not retrieve ${c,d,4,e}$. Precision is $2/4$ % % and recall is $3/4$. In a vector-based view, the same result could be % % represented as $[1,a,2,b,3,c,d,e,4]$ or as $[1,a,2,b,3,4,c,d,e]$. In % % the former case a user will have to go to document \#8 to find the % % last relevant document, in the latter, she will find it at position % % 6. Clearly, the latter seems preferable to the former. But precision % % and recall do not convey that view. \end{document} % LocalWords: Krichel Brookville NY Mishchenko Sobolev Koptyuga Riijsbergen's % LocalWords: NEP RePEc Nisa Bakkalbasi Losee metadata pre Swets Brookes ccrr % LocalWords: jonswe berbroo roblos Lofop cr wilcoo nep xxx barrueco rerank co % LocalWords: servap paugin svm torjoa tomkri min dev rrrrrrr mac bec pbe geo % LocalWords: ure mon ecm eec mic com tra int ino env acc fmk ets cfn ifn reg % LocalWords: hpe cmp ene edu cba dge ent afr hea agr tid rmg cbe gth exp ltv % LocalWords: dcm cwa eff pke evo ind cis cdm cul ias mfd spo %%%> Ms. Ref. No.: IPM2414 %%%> Title: Information retrieval performance measures for a current %%%> awareness report composition aid %%%> Information Processing & Management %%%> %%%> Dear Dr. Krichel, %%%> %%%> I am happy to inform you that your paper ms, "Information retrieval %%%> performance measures for a current awareness report composition aid" %%%> has been provisionally accepted for publication. However, the %%%> referee recommended minor revisions. We will consider the paper for %%%> final acceptance after the revision. %%% %%%> For your guidance, reviewers' comments are appended below. %%%> %%%> %%%> Reviewer #1: Page3 Para4 Line1: "included" or "relevant" %%% %%% fix #2 %%% %%%> Page3 Para5: not clear what the sentence "Thus, just looking at ?.." %%%> means %%% %%% fix #3 %%% %%%> Page5 Para2: "the measure should be 0" ? expected value of the %%%> measure should be 0" %%% %%% fix #6 %%% %%%> Page7 Para3: not clear what the sentence "thus, all documents?." %%%> Means %%% %%% fix #8, reformulation %%% %%%> Page8 Para2: Rephrase the sentence "According to?.." %%% %%% fix #9, removed words that should not have been there %%% %%%> Page9 Para1: divided by "n" or "r"? %%% %%% No. If we look at the best outcome, it has the relevant %%% outcomes at the tippy top, ok? So we count 1,2,...,r, and %%% need to find the average. The average of subsequent %%% natural numbers is the first plus the last divided by 2. %%% For the worst outcome, a similar reasoning applies. %%% %%%> Page10 : move Table 2 below the title 4.3 %%% %%% done %%% %%%> Page11: move Table 3 below the title 4.4 %%% %%% done %%% %%%> Page12 Footmark3 : Not sure if the geographical names help %%%> avoid this confusion! %%% %%% The other referee raised this too, so I prepare this version %%% without the geographical names. %%% %%%> Page13: move Table4 below the title 4.5 %%% %%% done %%% %%%> Page14: better if explain how and why the equation for Ob's %%%> measure was arrived at %%% %%% fix #12 %%% %%%> Page15: better if explain how and why the equation for Kolyma %%%> measure was arrive at %%% %%% fix #13 %%% %%%> Page17: better if an equation is added to show how the %%%> features are weighted/normalised %%% %%% oh well, I would not have thought this was necessary but I %%% have added it as fix #15 %%% %%%> Page17 Para1: not clear the selection of documents for the test %%%> and train sets in each iteration. Are they different in each run? %%% %%% They are. %%% %%%> Describe. %%% %%% I have added wording there and added one technical detail that %%% I previously thought was not worth including. Since the editor %%% does not seem to be concerned about containing the length of %%% the paper, I may as well write more. fix #16 %%% %%%> Page18 Last Para in Conclusions: How/Why do you say linear %%%> combination of two measures are the best. Where are the supporting %%%> evidence/data? Why/How did you arrive at vl+(i-v)k? %%% %%% ok. It's not conventional to have the main conclusion in the %%% actual conclusion, and here is burried in verbiage. fix #19 %%% %%%> Page18 results table: Label the leftmost column to indicate that it %%%> is about the "NEP report" %%%> %%%> %%%> Some Typos/English errors: %%%> Page3 Line1 : "such as" -> "in order to" %%% %%% fix #1 %%% %%%> Page3 one before the last line: "have the" repeated %%% %%% fix #4 %%% %%%> Page4 Line3 from the bottom: "are" is repeated %%% %%% fix #5 %%% %%%> Page4 Line2 from the bottom "means" %%% %%% fix #5 %%% %%%> Page6 Para1 Line5: "or" repeated %%% %%% fix #7 %%% %%%> Page8 Footmark Line1: "to" is repeated %%% %%% fix #10 %%% %%%> Page17 Para2 Line3: "that" -> "then" %%% %%% fix #11 %%% %%%> Page17 Para3 Line4: "able lift" -> "able to lift" %%% %%% fix #14 %%% %%%> Page17 Para3 Line5: "is" repeated %%% %%% fix #14 %%% %%%> Reviewer #2: This is an interesting manuscript. I recommend it be %%%> published after several issues have been addressed. %%%> %%%> There are no citations in sections 2 and 3. The author needs to %%%> address how some of the points here are addressed by others. In %%%> particular, the "requirements" in section 2 should be linked to %%%> other lists of requirements for retrieval measures. Rigorous metric %%%> based articles include Van Rijsbergen, JDoc, 30 (4) 365-373 and %%%> Shaw, JASIS 37 (5) 346-348. %%% %%% The work in both of these references is based on precision %%% and recall values. By the time section 2 starts, I already %%% have rejected this approach, and none of the two referees %%% found any problems with this. However, I have included %%% references to these two papers at the point in section %%% 1 where I say that I will not pursue this approach further. %%% fix #30 %%% %%%> Works by Bollman and (more recently) %%%> Dominich might be useful. %%% %%% I found Dominich's work, there is nothing in there that would %%% be interesting for my probelm. I did not see Bollmann-Sdorra's %%% work, but the way it is cited in Dominich, does suggest it %%% follows the same general approach as Van Rijsbergen and Shaw, %%% i.e. a set-based approach. In this paper, I use a general %%% approach that was used by Cooper and has seems largely have been %%% forgotten, that is to evaluate the information system by %%% the utility that it brings to its users. In most cases the %%% utility of the system is too vague to serve as a basic starting %%% point, but in my case it is reasonably straightforward. %%% %%%> Better names for the models would help - using the names of rivers %%%> or girlfriends or puppies is silly. Use explanatory names; do you %%%> find "Edith" or "Expected Search Length" to be easier to understand? %%% %%% It's an easy suggestion for a reviewer to make. It's also easy %%% to find descriptive names but they tend to be uncomfortably %%% long. I therefore have resorted to long names but use acronyms %%% in the discussion of the measures. If the referree can come %%% with names that are short enough to be used fully in the text %%% I will rather pleased. fixes #21 to #26 %%% %%%> When discussing precision and recall, as well as other measures such %%%> as ASL and ESL, remember that one can compute any of these measures %%%> for all or just the first part of a dataset, and one may compute the %%%> chosen measure at any given point in the set of documents being %%%> examined, such as computing the precision and recall after the first %%%> two documents have been retrieved. %%% %%% I am actually aware of this. I could mention this in the %%% paper, but it really is either to state that I know the %%% very basics of information retrieval, or to educate the %%% reader about them. I don't see how this improves %%% the problems with the precision and recall in this setting. %%% Maybe the reviewer knows, I would be interested. %%% What I propose in this paper is to move from set-based to %%% vector-based measures. Surly, if you can vary the sets along %%% your vector %%% of results. Presumably, if either precision or recall are %%% completely known at the all level of the size of the outcome, %%% the outcome vector can be determined. But a sequence of sets %%% strikes me a cumbersome kludge for a vector. %%% %%%> Avoid colloquialisms ("To find the expected value is *really* easy") %%% %%% fix #27 %%% %%%> as well as very subjective statements ("easy"). Be precise when %%%> possible ("which is not even close" [to what?]). %%% %%% both numbers are not close, fix #29 %%% %%%> Remember to end each sentence with a period, which may occur after %%%> a formula (put differently, the author often leaves out periods %%%> after formulas). %%% %%% I personnally find periods in displayed math an aberration, but %%% I don't want to make an issue of my personal preferences. Therefore %%% I added 23 periods and 4 commas to the manuscript. These changes %%% are not marked up in the source code. %%% %%%> More interpretation in the "Test" section of specific numbers and %%%> related comparisons and discussions would help. %%% %%% There results did raise a few eyebrows in the user community, and %%% interesting things could be written about them, but it is somewhat %%% out of scope for the purpose of this paper, because the paper is %%% about methods evaluation, not about the results of the evaluation in %%% a particular operational context, about which much more work could %%% be done, because it is a very interesting dataset and service. %%% But I did write a few more sentence about the results, limiting %%% myself, however, to concerns that strictly fall within the %%% remit of the paper. %%% %%%> With improved nomenclature and discussions of specific results, %%%> readers will find the article far more useful. %%% %%% I absolutely agree that the paper as it is now has improved %%% a lot over the previous version. %%%