Regular Expressions

multiFEED's URL Substitution feature lets you alter the article URL obtained from a feed before the body is opened. In most cases simple substitution is adequate to the task, but occasionally more sophisticated pattern matching and replacement is needed. Also, multiFEED allows you to reformat dates from RSS and Atom feeds on the fly if the publisher formats them incorrectly. For these tasks multiFEED provides Regular Expressions (often abbreviated as RegExp).

Writing Regular Expressions is an arcane skill, one far too complicated to teach here. The basic concepts are fairly easy to grasp, but becoming an expert can take years. Expertise with RegExps can mean the difference between an expression that requires fifty or a hundred characters to accomplish a complicated match and takes half a minute to execute and one that does the same thing in fifteen characters and does the job in less than a second. An elegantly designed RegExp can be a work of art. The good news is that you probably won't ever need monster Regular Expressions to manipulate URLs in multiFEED.

The basic trick with RegExp substitutions is to match one or more segments of a string even if they aren't always in the same place or contain the same text. Matched segments can then be put back together in any order and with any additional text you like. Regular Expressions grew out of the early Unix, C, and Perl environment, which is probably why the syntax is so convoluted. Although there is some variation in syntax between implementations, the most common are based on Perl.

multiFEED supports two pattern matching syntaxes:

RegExp
RegExp (greedy)

Both are Perl-like regular expression patterns. The difference is explained later in this topic.

multiFEED's Regular Expression processing uses the BlackBerry 10 RegExp library. The following is excerpted and adapted from their website regarding their implementation of Regular Expressions.

RegExps are built up from expressions, quantifiers, and assertions. The simplest expression is a character, e.g. x or 5. An expression can also be a set of characters enclosed in square brackets. [ABCD] will match an A or a B or a C or a D. We can write this same expression as [A-D], and an expression to match any capital letter in the English alphabet is written as [A-Z].

A quantifier specifies the number of occurrences of an expression that must be matched. x{1,1} means match one and only one x. x{1,5} means match a sequence of x characters that contains at least one x but no more than five.

Note that in general RegExps cannot be used to check for balanced brackets or tags. For example, a RegExp can be written to match an opening html and its closing , if the tags are not nested, but if the tags are nested, that same RegExp will match an opening tag with the wrong closing . For the fragment bold bolder, the first would be matched with the first , which is not correct. However, it is possible to write a RegExp that will match nested brackets or tags correctly, but only if the number of nesting levels is fixed and known. If the number of nesting levels is not fixed and known, it is impossible to write a RegExp that will not fail.

Suppose we want a RegExp to match integers in the range 0 to 99. At least one digit is required, so we start with the expression [0-9]{1,1}, which matches a single digit exactly once. This RegExp matches integers in the range 0 to 9. To match integers up to 99, increase the maximum number of occurrences to 2, so the RegExp becomes [0-9]{1,2}. This RegExp satisfies the original requirement to match integers from 0 to 99, but it will also match integers that occur in the middle of strings. If we want the matched integer to be the whole string, we must use the anchor assertions, ^ (caret) and $ (dollar). When ^ is the first character in a RegExp, it means the RegExp must match from the beginning of the string. When $ is the last character of the RegExp, it means the RegExp must match to the end of the string. The RegExp becomes ^[0-9]{1,2}$. Note that assertions, e.g. ^ and $, do not match characters but locations in the string.

If you have seen RegExps described elsewhere, they may have looked different from the ones shown here. This is because some sets of characters and some quantifiers are so common that they have been given special symbols to represent them. [0-9] can be replaced with the symbol \d. The quantifier to match exactly one occurrence, {1,1}, can be replaced with the expression itself, i.e., x{1,1} is the same as x. So our 0 to 99 matcher could be written as ^\d{1,2}$. It can also be written ^\d\d{0,1}$, i.e., From the start of the string, match a digit, followed immediately by 0 or 1 digits. In practice, it would be written as ^\d\d?$. The ? is shorthand for the quantifier {0,1}, i.e., 0 or 1 occurrences. ? makes an expression optional. The RegExp ^\d\d?$ means "From the beginning of the string, match one digit, followed immediately by 0 or 1 more digit, followed immediately by end of string".

To write a RegExp that matches one of the words 'mail' or 'letter' or 'correspondence' but does not match words that contain these words, e.g., 'email', 'mailman', 'mailer', and 'letterbox', start with a RegExp that matches 'mail'. Expressed fully, the RegExp is m{1,1}a{1,1}i{1,1}l{1,1}, but because a character expression is automatically quantified by {1,1}, we can simplify the RegExp to mail, i.e.,, an 'm' followed by an 'a' followed by an 'i' followed by an 'l'. Now we can use the vertical bar |, which means or, to include the other two words, so our RegExp for matching any of the three words becomes mail|letter|correspondence. Match 'mail' or 'letter' or 'correspondence'. While this RegExp will match one of the three words we want to match, it will also match words we don't want to match, e.g., 'email'. To prevent the RegExp from matching unwanted words, we must tell it to begin and end the match at word boundaries. First we enclose our RegExp in parentheses, (mail|letter|correspondence). Parentheses group expressions together, and they identify a part of the RegExp that we wish to capture. Enclosing the expression in parentheses allows us to use it as a component in more complex RegExps. It also allows us to examine which of the three words was actually matched. To force the match to begin and end on word boundaries, we enclose the RegExp in \b word boundary assertions: \b(mail|letter|correspondence)\b. Now the RegExp means: Match a word boundary, followed by the RegExp in parentheses, followed by a word boundary. The \b assertion matches a position in the RegExp, not a character. A word boundary is any non-word character, e.g., a space, newline, or the beginning or ending of a string.

If we want to replace ampersand characters with the HTML entity &, the RegExp to match is simply &, But this RegExp will also match ampersands that have already been converted to HTML entities. We want to replace only ampersands that are not already followed by amp;. For this, we need the negative lookahead assertion, (?!__). The RegExp can then be written as &(?!amp;), i.e., Match an ampersand that is not followed by amp;.

If we want to count all the occurrences of 'Eric' and 'Eirik' in a string, two valid solutions are \b(Eric|Eirik)\b and \bEi?ri[ck]\b. The word boundary assertion '\b' is required to avoid matching words that contain either name, e.g. 'Ericsson'. Note that the second RegExp matches more spellings than we want: 'Eric', 'Erik', 'Eiric' and 'Eirik'.

Characters and Abbreviations for Sets of Characters

Element	Meaning
c	A character represents itself unless it has a special RegExp meaning. e.g. c matches the character c.
\c	A character that follows a backslash matches the character itself, except as specified below. e.g., To match a literal caret at the beginning of a string, write \^.
\a	Matches the ASCII bell (BEL, 0x07).
\f	Matches the ASCII form feed (FF, 0x0C).
\n	Matches the ASCII line feed (LF, 0x0A, Unix newline).
\r	Matches the ASCII carriage return (CR, 0x0D).
\t	Matches the ASCII horizontal tab (HT, 0x09).
\v	Matches the ASCII vertical tab (VT, 0x0B).
\xhhhh	Matches the Unicode character corresponding to the hexadecimal number hhhh (between 0x0000 and 0xFFFF).
\0ooo	(i.e.,, \zero ooo) Matches the ASCII/Latin1 character for the octal number ooo (between 0 and 0377).
.	(dot) Matches any character (including newline).
\d	Matches a digit.
\D	Matches a non-digit.
\s	Matches a whitespace character.
\S	Matches a non-whitespace character.
\w	Matches a word character (letter, number, underline, Unicode Marks (Mn, Mc, Me)).
\W	Matches a non-word character.
\n	The n-th backreference, e.g. \1, \2, etc.

Sets of Characters

Square brackets mean match any character contained in the square brackets. The character set abbreviations described above can appear in a character set in square brackets. Except for the character set abbreviations and the following two exceptions, characters do not have special meanings in square brackets.

^	The caret negates the character set if it occurs as the first character (i.e., immediately after the opening square bracket). [abc] matches 'a' or 'b' or 'c', but [^abc] matches anything but 'a' or 'b' or 'c'.
-	The dash indicates a range of characters. [W-Z] matches 'W' or 'X' or 'Y' or 'Z'.

Using the predefined character set abbreviations is more portable than using character ranges across platforms and languages. For example, [0-9] matches a digit in Western alphabets but \d matches a digit in any alphabet.

Note: In other RegExp documentation, sets of characters are often called "character classes".

Quantifiers

By default, an expression is automatically quantified by {1,1}, i.e., it should occur exactly once. In the following list, E stands for expression. An expression is a character, or an abbreviation for a set of characters, or a set of characters in square brackets, or an expression in parentheses.

E?	Matches zero or one occurrences of E. This quantifier means The previous expression is optional, because it will match whether or not the expression is found. E? is the same as E{0,1}. e.g., dents? matches 'dent' or 'dents'.
E+	Matches one or more occurrences of E. E+ is the same as E{1,}. e.g., 0+ matches '0', '00', '000', etc.
E*	Matches zero or more occurrences of E. It is the same as E{0,}. The * quantifier is often used in error where + should be used. For example, if \s$ is used in an expression to match strings that end in whitespace, it will match every string because \s$ means Match zero or more whitespaces followed by end of string. The correct RegExp to match strings that have at least one trailing whitespace character is \s+$.
E{n}	Matches exactly n occurrences of E. E{n} is the same as repeating E n times. For example, x{5} is the same as xxxxx. It is also the same as E{n,n}, e.g. x{5,5}.
E{n,}	Matches at least n occurrences of E.
E{,m}	Matches at most m occurrences of E. E{,m} is the same as E{0,m}.
E{n,m}	Matches at least n and at most m occurrences of E.

To apply a quantifier to more than just the preceding character, use parentheses to group characters together in an expression. For example, tag+ matches a 't' followed by an 'a' followed by at least one 'g', whereas (tag)+ matches at least one occurrence of 'tag'.

Note: Quantifiers are normally "greedy". They always match as much text as they can. For example, 0+ matches the first zero it finds and all the consecutive zeros after the first zero. Applied to '20005', it matches '000'.

Capturing Text

Parentheses allow us to group elements together so that we can quantify and capture them. For example if we have the expression mail|letter|correspondence that matches a string we know that one of the words matched but not which one. Using parentheses allows us to "capture" whatever is matched within their bounds, so if we used (mail|letter|correspondence) and matched this RegExp against the string "I sent you some email" we can use use the matched string in the replacement expression.

We can use captured text within the RegExp itself. To refer to the captured text we use backreferences which are indexed from 1. For example we could search for duplicate words in a string using \b(\w+)\W+\1\b which means match a word boundary followed by one or more word characters followed by one or more non-word characters followed by the same text as the first parenthesized expression followed by a word boundary.

If we want to use parentheses purely for grouping and not for capturing we can use the non-capturing syntax, e.g. (?:green|blue). Non-capturing parentheses begin '(?:' and end ')'. In this example we match either 'green' or 'blue' but we do not capture the match so we only know whether or not we matched but not which color we actually found. Using non-capturing parentheses is more efficient than using capturing parentheses since the RegExp engine has to do less book-keeping.

Both capturing and non-capturing parentheses may be nested.

For historical reasons, quantifiers (e.g. *) that apply to capturing parentheses are more "greedy" than other quantifiers. For example, a*(a*) will match "aaa" when you might expect it to only match the final "a". This behavior is different from what other RegExp engines do (notably, Perl). To obtain a more intuitive capturing behavior, set substitution type to RegExp (greedy) for URL substitution, rather than the RegExp option.

Assertions

Assertions make some statement about the text at the point where they occur in the RegExp but they do not match any characters. In the following list E stands for any expression.

^	The caret signifies the beginning of the string. If you wish to match a literal ^ you must escape it by writing \\^. For example, ^#include will only match strings which begin with the characters '#include'. (When the caret is the first character of a character set it has a special meaning, see Sets of Characters.)
$	The dollar signifies the end of the string. For example \d\s*$ will match strings which end with a digit optionally followed by whitespace. If you wish to match a literal $ you must escape it by writing \\$.
\b	A word boundary. For example the RegExp \bOK\b means match immediately after a word boundary (e.g. start of string or whitespace) the letter 'O' then the letter 'K' immediately before another word boundary (e.g. end of string or whitespace). But note that the assertion does not actually match any whitespace so if we write (\bOK\b) and we have a match it will only contain 'OK' even if the string is "It's OK now".
\B	A non-word boundary. This assertion is true wherever \b is false. For example if we searched for \Bon\B in "Left on" the match would fail (space and end of string aren't non-word boundaries), but it would match in "tonne".
(?=E)	Positive lookahead. This assertion is true if the expression matches at this point in the RegExp. For example, const(?=\s+char) matches 'const' whenever it is followed by 'char', as in 'static const char '. (Compare with const\s+char, which matches 'static const char '.)
(?!E)	Negative lookahead. This assertion is true if the expression does not match at this point in the RegExp. For example, const(?!\s+char) matches 'const' except when it is followed by 'char'.