Publication Date Transformation

As of version 2.8.0, multiFEED allows you to transform invalid date strings from feeds into valid RFC3339 formatted dates so multiFEED can read them correctly. Doing this requires some skill writing regular expressions.

There are three parts to an RFC3339 date string...

• The date (yyyy-mm-dd)
• The time (hh:mm:ss)
• The offset from UTC ([+/-]hhmm) OR the letter 'Z' to indicate that the date/time is UTC/GMT

A properly formatted RFC3339 date looks like this...

<date>T<time>Z

or...

<date>T<time><offset>

The date should include either 'Z', or an offset, not both. Here are a few real examples...

• 2018-01-15T13:15:30Z (1:15:30pm GMT on January 15th, 2018)
• 2017-06-20T10:30:00-0500 (10:30am on June 20th, 2017 with a timezone offset of -5 hours)
• 2005-12-25 01:00:00+0000 (1am GMT on Christmas morning 2005)

In that last example the offset is +0000, which is the same as if 'Z' was used instead. Either one is valid. Also note that there is no 'T' between the date and time portions of the date string. Although it is traditional to place the 'T' there, it is valid to use a space instead.

To transform an invalid date you must write a regular expression which captures the various parts of an RFC3339 date from the date string in the feed, and then paste them all back together into a valid RFC3339 date string. Here's a real-life example from an actual published feed...

For some reason, the Let's Encrypt RSS feed suddenly changed the format of the article dates from a valid RFC822 format to a format neither RFC822 nor RFC3339, causing multiFEED to start reading the article dates incorrectly. A sample of this invalid date format is...

2018-04-04 16:00:00 +0000 UTC

As you can see, it is similar to RFC3339, but there is an extra space between the time and offset fields and the extra characters ' UTC' tacked on the end. Unfortunately this is enough of a difference that the date parser in multiFEED is unable to figure out what date and time it is describing. The first task is writing the regular expression to extract the required parts for the transformation. Regular expressions are very powerful and there are often many ways to accomplish the same task, but this one will work...

(\d{4}-\d\d-\d\d)\s+(\d\d-\d\d-\d\d)\s+([-\+]\d{4})\s+UTC

This is not a tutorial on regular expressions, but here is a simplified explanation of what this expression does...

• (\d{4}-\d\d-\d\d) - Match exactly four digits in a row followed by a hyphen followed by two digits followed by a hyphen followed by two digits. The parenthesis around this part of the expression means save the found digits and hyphens to use later (as '\1').
• \s+ - Match one or more spaces or other whitespace. No parenthesis around it so just match it, don't save it for later.
• (\d\d-\d\d-\d\d) -  Match two digits in a row followed by a hyphen followed by two digits followed by a hyphen followed by two digits. The parenthesis around this part of the expression means save the found digits and hyphens to use later (as '\2').
• \s+ - Match one or more spaces or other whitespace. No parenthesis around it so just match it, don't save it for later.
• ([-\+]\d{4}) - Match a '-' or '+' followed by exactly four digits. The parenthesis around this part of the expression means save the sign and found digits to use later (as '\3'). The '+' character normally has a special meaning in regular expressions so it is preceded by '\' in this expression to indicate it is to be used as a literal '+'.
• \s+UTC - Match one or more spaces or other whitespace, followed by the characters 'UTC'. No parenthesis around it so just match it, don't save it for later.

Running this regular expression against the invalid date grabs three substrings, the date, the time, and the offset (with sign). The rest of the invalid date string is discarded. Now lets write the transformation string to recombine all three parts into a valid RFC3339 date...

\1T\2\3

That's it! This transformation string outputs the '2018-04-04' followed by a 'T' and then the '16:00:00' immediately followed by the '+0000' offset...

2018-04-04T16:00:00+0000

... which is a valid RFC3339 date format. Entering this regular expression and transformation string into multiFEED allows it to successfully read the date and time of articles in the Let's Encrypt feed. This example was fairly simple since the invalid date was quite close to a valid format, but regular expressions are powerful enough to correctly transform even radically different date/time formats.

The substrings captured by the regular expression are referenced in the transformation string with a backslash and the ordinal number of the capture, counting from left to right. For instance, '\5' refers to the fifth substring captured by the regular expression. As noted, parenthesis are used to indicate the groups of characters to be captured. For more information on the syntax of regular expressions understood by multiFEED, see here.