Regular Expression Syntax

Introduction

Example

> se *.txt -regex \b(\w+)((?:\s)+)\1\b

Very briefly:

• The regular expressions use Perl syntax.
• All character groups, such as letters or punctuation, are defined according the Latin 1 (ISO 8859-1) character set.
  

For a good and freely available tutorial on regular expressions, I would recommend http://www.regular-expressions.info/tutorial.html

Boost Regex Syntax

The following description of the syntax for regular expressions originally comes from the Boost Regex documentation. It has been adapted here to reflect how it is being used in the Glindra se command.

You can find the original version of the document, as well as additional information about the Boost Regex engine, at the Boost site.

Literals

All characters are literals except: .|*?+(){}[]^$\ These characters are literals when preceded by a \. A literal is a character that matches itself.

If the search is done with the -noexact_case option (which is the default), letters will also match their upper/lower case counterpart. This applies both to a-zA-Z and to the national letters in the Latin 1 (ISO 8859-1) character set, like éñüöÉÑÜÖ.

Wildcard

The dot character . matches any single character.

Repeats

A repeat is an expression that is repeated an arbitrary number of times.

An expression followed by * can be repeated any number of times including zero.

An expression followed by + can be repeated any number of times, but at least once.

An expression followed by ? may be repeated zero or one times only.

When it is necessary to specify the minimum and maximum number of repeats explicitly, the bounds operator {} may be used, thus a{2} is the letter a repeated exactly twice, a{2,4} represents the letter a repeated between 2 and 4 times, and a{2,} represents the letter a repeated at least twice with no upper limit. Note that there must be no white-space inside the {}, and there is no upper limit on the values of the lower and upper bounds.

All repeat expressions refer to the shortest possible previous sub-expression: a single character; a character set, or a sub-expression grouped with () for example.

Examples:

ba* will match all of b, ba, baaa etc.

ba+ will match ba or baaaa for example but not b.

ba? will match b or ba.

ba{2,4} will match baa, baaa and baaaa.

Non-greedy repeats

Non-greedy repeats are possible by appending a ? after the repeat; a non-greedy repeat is one which will match the shortest possible string.

For example to match html tag pairs one could use something like:

<\s*tagname[^>]*>(.*?)<\s*/tagname\s*>

In this case $1 will contain the text between the tag pairs, and will be the shortest possible matching string. 

Parenthesis

Parentheses serve two purposes, to group items together into a sub-expression, and to mark what generated the match. For example the expression (ab)* would match all of the string ababab. In the example $1 would denote the final ab of the matching string. It is permissible for sub-expressions to match null strings. If a sub-expression takes no part in a match - for example if it is part of an alternative that is not taken - then both of the iterators that are returned for that sub-expression point to the end of the input string, and the matched parameter for that sub-expression is false. Sub-expressions are indexed from left to right starting from 1, sub-expression 0 is the whole expression.

Non-Marking Parenthesis

Sometimes you need to group sub-expressions with parenthesis, but do not want the parenthesis to spit out another marked sub-expression, in this case a non-marking parenthesis (?:expression) can be used. For example the following expression creates no sub-expressions:

(?:abc)*

Forward Lookahead Asserts 

There are two forms of these; one for positive forward lookahead asserts, and one for negative lookahead asserts:

(?=abc) matches zero characters only if they are followed by the expression abc.

(?!abc) matches zero characters only if they are not followed by the expression abc.

Independent sub-expressions

(?>expression) matches expression as an independent atom (the algorithm will not backtrack into it if a failure occurs later in the expression).

Alternatives

Alternatives occur when the expression can match either one sub-expression or another, each alternative is separated by a |. Each alternative is the largest possible previous sub-expression; this is the opposite behavior from repetition operators.

Examples:

a(b|c) could match ab or ac.

abc|def could match abc or def.

Sets

A set is a set of characters that can match any single character that is a member of the set. Sets are delimited by [ and ] and can contain literals, character ranges, character classes, collating elements and equivalence classes. Set declarations that start with ^ contain the complement of the elements that follow.

Examples:

Character literals:

[abc] will match either of a, b, or c.

[^abc] will match any character other than a, b, or c.

Character ranges:

[a-z] will match any character in the range a to z.

[^A-Z] will match any character other than those in the range A to Z.

Character classes are denoted using the syntax [:classname:] within a set declaration, for example [[:space:]] is the set of all whitespace characters. 

The character classes are defined according to Latin 1 (ISO 8859-1), which means that the national letters and punctuation characters in the upper half of the 8 bit code are correctly classified. The character set is not affected by the locale setting, so the program will work the same whether it is being run on a computer inside the US or elsewhere.

The available character classes are:

alnum	Any alpha numeric character. [[:alnum:]] is the same as [[:alpha:][:digit:]].
alpha	Any alphabetical character. [[:alpha:]] is the same as [[:upper:][:lower:]].
blank	Any blank character, either a space \040 or a tab \t. Note that the non-breaking space character \0240 is not included in this character class.
cntrl	Any control character.
digit	Any digit 0-9.
graph	Any graphical character.
lower	Any lower case character. This includes the English letters a-z, as well as the national letters ßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ
print	Any printable character.
punct	Any punctuation character.
space	Any whitespace character, either space \040, tab \t, newline \n, vertical tab \v, form feed \f, or carriage return \r. Note that the non-breaking space character \0240 is not included in this character class.
upper	Any upper case character. This includes the English letters A-Z, as well as the national letters ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞ
xdigit	Any hexadecimal digit character, 0-9, a-f and A-F.
word	Any word character - all alphanumeric characters plus the underscore. [[:word:]] is the same as [[:alnum:]_]. Note that this includes the national letters, and not just a-zA-Z.

A table that shows the character classes for each of the 256 different Latin 1 characters can be found here.

There are some shortcuts that can be used in place of the character classes, provided the flag regex_constants::escape_in_lists is set then you can use:

\w in place of [:word:]

\s in place of [:space:]

\d in place of [:digit:]

\l in place of [:lower:]

\u in place of [:upper:] 

Collating elements take the general form [.tagname.] inside a set declaration, where tagname is either a single character, or a name of a collating element, for example [[.a.]] is equivalent to [a], and [[.comma.]] is equivalent to [,]. The library supports all the standard POSIX collating element names, and in addition the following digraphs: ae, ch, ll, ss, nj, dz, lj, each in lower, upper and title case variations. Multi-character collating elements can result in the set matching more than one character, for example [[.ae.]] would match two characters, but note that [^[.ae.]] would only match one character. 

To include a literal - in a set declaration then: make it the first character after the opening [ or [^, the endpoint of a range, a collating element, or precede with an escape character as in [\-]. To include a literal [ or ] or ^ in a set then make them the endpoint of a range, a collating element, or precede with an escape character.

Line anchors

An anchor is something that matches the null string at the start or end of a line: ^ matches the null string at the start of a line, $ matches the null string at the end of a line.

Back references

A back reference is a reference to a previous sub-expression that has already been matched, the reference is to what the sub-expression matched, not to the expression itself. A back reference consists of the escape character \ followed by a digit 1 to 9, \1 refers to the first sub-expression, \2 to the second etc. For example the expression (.*)\1 matches any string that is repeated about its mid-point for example abcabc or xyzxyz. A back reference to a sub-expression that did not participate in any match, matches the null string: NB this is different to some other regular expression matchers.

Characters by code

This is an extension to the algorithm that is not available in other libraries, it consists of the escape character followed by the digit 0 followed by the octal character code. For example \023 represents the character whose octal code is 23. Where ambiguity could occur use parentheses to break the expression up: \0103 represents the character whose code is 103, (\010)3 represents the character 10 followed by 3. To match characters by their hexadecimal code, use \x followed by a string of hexadecimal digits, optionally enclosed inside {}, for example \xf0 or \x{f0}.

Word operators

The following operators are provided for compatibility with the GNU regular expression library.

\w matches any single character that is a member of the "word" character class, this is identical to the expression [[:word:]].

\W matches any single character that is not a member of the "word" character class, this is identical to the expression [^[:word:]].

\< matches the null string at the start of a word.

\> matches the null string at the end of the word.

\b matches the null string at either the start or the end of a word.

\B matches a null string within a word.

The start of the line is considered to be a potential start of a word, and the end of the line is considered to be a potential end of a word.

Buffer (line) operators

The following operators are provided for compatibility with the GNU regular expression library, and Perl regular expressions:

\` matches the start of a line.

\A matches the start of the line.

\' matches the end of a line.

\z matches the end of a line.

\Z matches the end of a line, or possibly one or more new line characters followed by the end of the line.

Escape operator

The escape character \ has several meanings.

Inside a set declaration, whatever follows the escape is a literal character regardless of its normal meaning.

The escape operator may introduce an operator for example: back references, or a word operator.

The escape operator may make the following character normal, for example \* represents a literal * rather than the repeat operator.

Single character escape sequences

The following escape sequences are aliases for single characters:
 

Escape sequence	Character code	Meaning
\a	0x07	Bell character.
\f	0x0C	Form feed.
\n	0x0A	Newline character.
\r	0x0D	Carriage return.
\t	0x09	Tab character.
\v	0x0B	Vertical tab.
\e	0x1B	ASCII Escape character.
\0dd	0dd	An octal character code, where dd is one or more octal digits.
\xXX	0xXX	A hexadecimal character code, where XX is one or more hexadecimal digits.
\x{XX}	0xXX	A hexadecimal character code, where XX is one or more hexadecimal digits, optionally a Unicode character.
\cZ	z-@	An ASCII escape sequence control-Z, where Z is any ASCII character greater than or equal to the character code for '@'.

Miscellaneous escape sequences:

The following are provided mostly for perl compatibility, but note that there are some differences in the meanings of \l, \L, \u, and \U:
 

\w	Equivalent to [[:word:]].
\W	Equivalent to [^[:word:]].
\s	Equivalent to [[:space:]].
\S	Equivalent to [^[:space:]].
\d	Equivalent to [[:digit:]].
\D	Equivalent to [^[:digit:]].
\l	Equivalent to [[:lower:]].
\L	Equivalent to [^[:lower:]].
\u	Equivalent to [[:upper:]].
\U	Equivalent to [^[:upper:]].
\C	Any single character, equivalent to '.'.
\Q	The begin quote operator, everything that follows is treated as a literal character until a \E end quote operator is found.
\E	The end quote operator, terminates a sequence begun with \Q.