Corpus Query with Regular Expressions: Introduction to CQPWeb

Background

The Corpus Workbench (CWB) is a collection of open-source tools for managing and querying large text corpora (ranging from 10 million to 2 billion words) with linguistic annotations. Its central component is the Corpus Query Processor (CQP) which allows to query language corpora in an efficient way using pattern matching with regular expressions.

CQPweb is the web-based graphical user interface (GUI) for some elements of the CWB, in particular the query processor CQP.

In this tutorial we will learn how to query corpora using CQP and its graphical user interface using the CQPweb server at Saarland University.

CQPweb has two query language options simple query and CQP syntax. In this tutorial we will use the CQP syntax only, because

• it is not really more complicated than the simple query
• you have more options available
• you can use the same queries on the command line

Getting started

• you need to register to the CQPweb installation of the UdS. Be aware that all CQPweb installations have their own user management, your user account of another CQPweb (i.e., the CQPweb at Lancaster) will not be valid.
• the registration process is mostly automatic, users are assigned rights based on their email address. Thus, in order to be granted the necessary rights please use:
  • your university email address or
  • the email address you used for registration to a workshop/course
• please also indicate your real name and affiliation (if applicable), this makes it easier to identify your account in case of problems
• after registration you will get a confirmation email with a link to activate your account. It may happen that you do not get this email, e.g. because it does not pass the SPAM filter. In this case, please contact the administator.
• there is also a video tutorial of the registration process by Andrew Hardie (Lancaster University)

Basic queries

The standard setting in this CQPweb installation is Simple query as Query mode. In this tutorial, we use CQP Synax. So, before we get started, we need to change Query mode from Simple query to CQP syntax!

A word - or better token - in CQP Syntax is surrounded by square-brackets [].
Inside of these brackets you can specify the token more closely using an attribute-value syntax: [att="value"]

If you want to search for a word, e.g. the word word we use the attribute word with the value word:

[word="word"]

As a result we get concordance lines for all instances of the word word in the corpus. Executed on the BLOB corpus, the concordance should look like this.

A concordance is an alphabetical listing of the words in a text, given together with the contexts in which they appear. The most common form of concordance today is the Keyword-in-Context (KWIC) index, in which each word is centered in a fixed-length field (e.g., 80 characters).

For the attribute word - and only for this attribute - there is a second option, namely simply to enter the word in quotation marks:

"word"

The attribute word retrieves only the exact word form, i.e., if you want to look for the word word in its plural form words, the query would be:

[word="words"]

To look for both forms (word and words) at the same time, we have to use regular expressions, e.g. using the or-operator |

[word="word|words"]

or making the s at the end optional using a question mark ?

[word="words?"]

Another possibility, depending on the annotation of the corpus, is to use the attribute lemma (base form) with the value word. This gives us all word forms in the corpus that have the lemma (base form) word: word, words

[lemma="word"]

In order to look for all occurrences of a common noun in the singluar (given your text is tagged with the UPenn tagset) we can use the following query.

[pos="NN"]

Note that different tagsets may be used for different corpora. The menu item Corpus Info usually tells you which tagset has been used for the pos annotation of a corpus.

[word = "words?"] # final "s" is optional

[word="(re)?search"] # the sequence "re" is optional

[word = "words*"]

[word="word.?"]
[word="word.*"]

[word="[whm]orse"]

[word="[a-zA-Z]orse"]

[word="[^whm]orse"]

[word="im(portant|possible)"]
[lemma="research|investigation"]
[lemma="research" & pos="NN"]
[pos!="V.*|N.*"]