wisski_date_field_extractor

The WissKI date field stores what was typed and, on save, fills sibling fields with a time span Views can filter and sort.

To install the module and set up a field, see Installation and configuration.

What is stored¶

Qualified begin and qualified end are the stated date (1943-01-01, 1943-12-31). Earliest and latest begin and end are sortable integers, year * 10000 + month * 100 + day, including negative years.

1943 still stores qualified begin 1943-01-01 and qualified end 1943-12-31. February of a year-month such as 1944-02 still ends on the 28th in those sibling integers. That is the old Level 0 path, kept so existing Views stay as they are. The parser and the stored pathbuilder seconds use the real last day, so 2000-02 ends on 29 February there. The two results are a documented legacy difference, and new EDTF features are not added on the Level 0 path.

Widths¶

The approximate widths are set at EDTF widths. 1943~ with approximate years set to 1 is 1942 through 1944. Set to 10, it is 1933 through 1953. 1900? stays the year 1900. 1900% uses the same width as ~: 1899 through 1901, and it also records that 1900 may be wrong. EDTF names the kind of uncertainty. How many years, months or days that adds is this module's rule, and the numbers above are that rule. ? does not add a width. X ignores the widths: 201X is 2010 through 2019, and 190X is 1900 through 1909.

Saving an entity again does not reread an unchanged EDTF string, unless that value adjusts its own widths, so a width change needs drush wisski:edtf-reparse.

Time-span collection¶

The parsed value is a time-span collection. Each member is a time-span. The collection is stored on the dates field of the time expression: one reference per member, earliest first. One interval is one time-span. A set is one time-span per member. Which field receives which value is listed under Map existing bundles by hand.

EDTF profile¶

This is a WissKI profile of EDTF, not a claim that Level 0, 1 or 2 is fully supported. Level 0 date and time, including Z and a UTC offset, is outside the profile: the time is dropped and the calendar date in the string is kept. 1985-04-12T23:20:30+14:00 stays 12 April 1985. It is not shifted into another zone. Basic format without hyphens (20050924) is rejected. Seasons 25 to 41 are not accepted. 1670..1672 on its own is an extension; the EDTF interval is 1670/1672. Inside a set, .. is the EDTF list range. An open side is ... An unknown side is empty, on either side of /.

Sign	Meaning
`~`	Approximate. The approximate width is added on each side. The stated value is kept beside the widened seconds. A mark immediately right of a component covers that component and everything to its left (`2004-06~-11`). A mark immediately left of a component covers that component only (`2004-%06-11`)
`?`	Uncertain, stored under the reading `Probable`. That name does not state a probability. The stated date stays. It does not add a width
`%`	Both, and the same left/right rule as `~` and `?`. `2004-06-11%` covers year, month and day. `1900-%04` covers the month only
`X`	Unspecified. The whole range of that digit. Exactly one value in that range applies
`..` or `/`	Interval. `1670/1672` is one time-span that began in 1670 and ended in 1672. `1670..1672` alone is a WissKI extension of that. `../` and `/..` are open: the configured year is a sort index only, and that side has no second and no RDF bound. An empty side (`1984/` or `/1984`) is unknown and leaves the integer empty too
`!(…)`	At some time within. A WissKI extension, see Time model
`=(…)`	Exactly throughout. A WissKI extension, see Time model
`{a,b}`	All of these dates. Each member is its own time-span, and the time expression is typed `all_of`. A year between them does not match Within
`[a,b]`	Exactly one of these dates, typed `one_of`. Within still matches a listed date, because the value might be that one
`21`–`24`	Season, as a concrete span. `21` is 1 March–31 May, `22` is 1 June–31 August, `23` is 1 September–30 November, `24` is 1 December through the last day of February in the next year, including a leap day
`S`	Significant digits. `1950S2` is estimated as 1950, and the outer bounds are 1900 through 1999. It is not a certain span and not an unspecified digit
`-` or `Y`	Negative and extended years (`-0044`, `Y-12345`). A year whose packed day does not fit a signed 32-bit integer stays a parsed value: the text is kept, the integer fields stay empty, and that is not the same as text that is not EDTF

With the default widths, 1900-04%-XX runs from 1 March 1899 through 31 May 1901. % to the right of April covers the year and the month and adds the approximate width to each. The day stays unspecified. A time-span stores a known side as 00:00:00 and 23:59:59. An open .. side and an unknown / side store no second.

The sibling integer is year * 10000 + month * 100 + day in a signed 32-bit column, from -2147483648 to 2147483647. That is a Drupal integer column, not an EDTF limit. Y214748 still fits. Y214749 and Y-214749 do not. The original text is kept and the span fields are left empty. The value is still parsed. Text that is not EDTF is not.

Views¶

Within — one stored date overlaps the value. A year between two set members does not match: [1900,1904,=(1908)] does not match 1902–1903. This uses the date-list field, an unlimited text field with one value per date, and compares at most 64 dates. A longer value matches nothing. Without that field, Within matches nothing, because the first and last date alone would match a gap year.
Before (including the bound) — the last date ends on or before the value. It reads the latest-end integer.
After (including the bound) — the first date starts on or after the value. It reads the earliest-begin integer.

The earliest-begin integer is the first date, so it is also the sort key. Setup: Set up Views.

Views on the time model¶

A field that references the time expression bundle or the time-span bundle, such as Event → Date, gets a second Views entry, …: time-span bounds. It filters and sorts by the four bounds of the referenced time-spans, read through the mapped fields. The bundle that holds the field needs no path of its own down to the time-spans. The handler settings name the bounds by their field labels:

Bound	Default label	Filled with
Outer begin	Some time within, begin	the earliest second the time-span can begin
Inner begin	At least throughout, begin	the latest second it can begin
Inner end	At least throughout, end	the earliest second it can end
Outer end	Some time within, end	the latest second it can end

The filter value is EDTF. By default (All mapped bounds, by operator) the operator picks the bounds, so one filter covers what the four bounds say:

Operator	Matches a time-span that	Compares
Is before	surely ended before the value begins	Some time within end < first second
Is before or in	surely ended by the end of the value	Some time within end ≤ last second
Is in	lies entirely in the value	Some time within begin ≥ first second and end ≤ last second
Is in or after	surely began at or after the start of the value	Some time within begin ≥ first second
Is after	surely began after the value ends	Some time within begin > last second
Overlaps	may have happened at the same time	Some time within begin ≤ last second and end ≥ first second
Went on throughout	surely went on during the whole value	At least throughout begin ≤ first second and end ≥ last second

A comparison of two bounds reads the same time-span, so [1800,1850] does not overlap 1825. For Overlaps, a side the time-span leaves open is unbounded: after 1853 stores no end and overlaps 1800/1900, and 1900/.. overlaps every later value. The other operators need the bounds they compare. One matching time-span is enough. Went on throughout needs the inner bounds, which only an interval such as 1900/1910 or =(…) has. Has a time-span and Has no time-span take no value and count a time-span with any mapped bound. A single bound can be chosen instead: the operators before, in and after then compare that bound alone, and Overlaps and Went on throughout keep their two bounds.

The sort uses every mapped bound by default (All mapped bounds, chronologically). Ascending, the keys are Some time within begin, then Some time within end, then At least throughout begin, then At least throughout end: 1900 comes before 1900/1950, and =(1900/1910) before 1900/1910. The outer pair comes first because the inner bounds are empty for a single date and for !(…); ranked earlier, they would sort by whether they exist. Descending mirrors the time axis and starts with the end: the event that ended last comes first. A bound that is not mapped is left out. One bound can be chosen instead. Ascending uses the earliest value of all time-spans, descending the latest, so a set sorts by its first date upwards and by its last date downwards. An entity without a time-span comes first ascending and last descending.

Dates are compared as xsd:dateTime, so negative and five-digit years keep their order. The store has to cast a stored bound to xsd:dateTime. Each filter and each sort adds the reference path and the bound paths to the one SPARQL query of the view; there is no additional query per row. The sort reads all its bounds in one grouped sub-select. With 10,000 events in a local RDF4J native store, a page of 20 sorted by all bounds took about 0.25 s, with a filter about 0.3 to 0.45 s, plus up to 0.2 s for the pager's count.

Time model¶

A WissKI date says when something happened. A date field holds one EDTF value, and that value is read in one of three ways. All three place the event somewhere between 1 January 1900 and 31 December 1910. They differ in what is known about its begin and its end:

Typed	Begin	End	So it surely went on	Example
`!(1900/1910)`	some time 1900–1910	some time 1900–1910	unknown: it may have been short	A letter written at some time in that decade
`1900/1910`	in 1900	in 1910	1901 through 1909	A church begun in 1900 and finished in 1910
`=(1900/1910)`	on 1 January 1900	on 31 December 1910	the whole span, no longer	An exhibition open from 1 January 1900 to 31 December 1910

               1900   1901 … 1909   1910
!(1900/1910)   ·········?·········?·····   somewhere inside, length unknown
                        ███                (a few months in 1905 is possible)

1900/1910      ?██████████████████████?    begin somewhere in 1900,
               └1900┘             └1910┘   end somewhere in 1910,
                      ████████████         1901–1909 for sure

=(1900/1910)   ████████████████████████    exactly 1 Jan 1900 – 31 Dec 1910

Which one to type

You only know a period in which it happened at some time: !(A/B).
You know the year (or month, or day) it began and the one it ended: A/B.
You know the exact first and last day: =(A/B).

The difference only shows on an interval. A single date such as 1900 begins and ends in the same year, so nothing follows about when it surely went on: it may have been one day. 1900 and !(1900) are the same. Only =(1900) says more: the whole year.

1900/1910 is not =(1900/1910)

In EDTF, 1900 on the left of / is the precision of the begin, not 1 January. The begin can be 3 October 1900, and the end 5 March 1910. Only =(…) fixes the begin to the first second and the end to the last second.

"At least throughout, perhaps longer" has no sign.

!(…) and =(…) are WissKI extensions of EDTF. They enclose the whole value or one member of a set, never one side of /. Inside the brackets is EDTF. =(…) needs a known begin and end, so it rejects X, significant digits, an open or unknown side, and vor / nach. !(…) accepts all of them.

Sets and choices¶

{a,b} means every listed date applies, [a,b] exactly one of them. A value that is one date holds one time-span; a set holds one time-span per member, and a year between two members is not part of the value. Each member can carry its own mark:

Typed	Meaning
`{1900,1932,1934}`	It happened in each of these three years
`[1900,1904,=(1908)]`	It happened at some time in 1900, or at some time in 1904, or exactly during the whole of 1908
`=({1900,1904})`	It went on during the whole of 1900 and during the whole of 1904

Qualifiers on an interval¶

~, % and X widen the unit the begin or the end lies in, by the widths. The surely covered stretch between the two units shrinks by the same amount. With the default width of one year, 1900~/1910 began between 1899 and 1901 and surely went on from 1902 through 1909. When the begin's unit and the end's unit touch or overlap, as in 1900~/1901, no instant is surely covered. An open (..), unknown (empty), vor or nach side gives no such limit.

2004-06-~01/2004-06-~20 began between 31 May and 2 June 2004, ended between 19 and 21 June, and surely went on from 3 through 18 June.

What is written on save¶

With a saved mapping, from Add the WissKI time model or by hand, each date stores the four bounds and its text. The time expression stores the typed text and, for a set or a choice, the type. Without a saved mapping, nothing is written. A save whose text did not change does not write the linked dates again.

The four bounds already tell 1900/1910, !(1900/1910) and =(1900/1910) apart. Views compare the sibling integers and the date list, so all three readings filter and sort the same way.

RDF¶

The mapped pathbuilder fields are ordinary WissKI triples. The turtle below is what those fields mean. A qualifier stays in the EDTF text and in the widened bounds. How sure a time-span is, is in addition written on its four bound triples as RDF 1.2 statements: the time-span's Uncertainty field is an annotation field, so each bound triple gets a reifier of class wisski:Uncertainty_Annotation with P3 has note and the reading key (probable, approximate, unspecified, estimated). A certain time-span has none.

On the time expression, the dates field shows a * after each uncertain time-span. Its tooltip names the reading, for example Uncertainty: Uncertain, in the interface language. The star follows the mapped uncertainty field: a time-span without a reading, or a model without that field, shows none.

Structure¶

The dated event links by P4_has_time-span to a time expression. The time expression is the envelope around the value: it contains one E52_Time-Span per date by P86i_contains, also when the value is a single date. The typed text is an E41_Appellation of the time expression.

E5 Event
 └─ P4_has_time-span ─> Time_Expression        P82a, P82b of the whole value
     ├─ P2_has_type ──> all_of | one_of        only for {…} and […]
     ├─ P1_is_identified_by ─> E41 ── P190 "the typed EDTF"
     └─ P86i_contains ─> E52_Time-Span          one per date: P82a, P81a, P81b, P82b

Declare the time expression a subclass of E52

P4 and P86i are defined on E52_Time-Span. Declare Time_Expression as rdfs:subClassOf crm:E52_Time-Span in the ontology. A tool that knows only CIDOC CRM then reads the envelope as "at some time between the first and the last date", which is true for every reading.

The envelope carries P82a from the earliest begin and P82b from the latest end, and never P81. An open first or last member leaves that side out.

Vocabulary¶

Add the WissKI time model inserts two classes into the graph http://wiss-ki.eu/ontology/default/time-additions when the store lacks them. The bounds, the type and the appellation are CIDOC CRM 7.1.3; it has no E49 or P78 any more, so the text is an E41_Appellation by P1. Only a set or a choice has a type: all_of_these_dates (All of these dates) or one_of_these_dates (One of these dates), the name of a Time expression type that the pathbuilder shares between all time expressions with that key. One date or one interval leaves the type empty.

Stored field	Property	Use
Time expression	`http://wiss-ki.eu/ontology/default/Time_Expression`	The envelope, subclass of `E52`
Time expression type	`http://wiss-ki.eu/ontology/default/Time_Expression_Type`	Subclass of `E55`. Named `all_of_these_dates` or `one_of_these_dates`
Some time within, begin	`crm:P82a_begin_of_the_begin`	Not begun before this second
Some time within, end	`crm:P82b_end_of_the_end`	Not ended after this second
At least throughout, begin	`crm:P81a_end_of_the_begin`	Begun by this second at the latest
At least throughout, end	`crm:P81b_begin_of_the_end`	Ended at this second at the earliest
Contains	`crm:P86i_contains`	Time expression to each date
Has type	`crm:P2_has_type`	Time expression to its type
Identified by, Appellation, Symbolic content	`crm:P1_is_identified_by`, `crm:E41_Appellation`, `crm:P190_has_symbolic_content`	The typed text

crm: is http://www.cidoc-crm.org/cidoc-crm/. The examples below use wtime: for the time expression. The class the model inserts is http://wiss-ki.eu/ontology/default/Time_Expression.

Bounds per reading¶

P82a to P81a is the window the begin lies in, P81b to P82b the window of the end. Between P81a and P81b the value surely holds. The three readings fill the four bounds like this:

Typed	`P82a`	`P81a`	`P81b`	`P82b`
`1900`	1900-01-01T00:00:00	–	–	1900-12-31T23:59:59
`1900/1910`	1900-01-01T00:00:00	1900-12-31T23:59:59	1910-01-01T00:00:00	1910-12-31T23:59:59
`!(1900/1910)`	1900-01-01T00:00:00	–	–	1910-12-31T23:59:59
`=(1900)`	1900-01-01T00:00:00	1900-01-01T00:00:00	1900-12-31T23:59:59	1900-12-31T23:59:59
`=(1900/1910)`	1900-01-01T00:00:00	1900-01-01T00:00:00	1910-12-31T23:59:59	1910-12-31T23:59:59
`1900~/1910`	1899-01-01T00:00:00	1901-12-31T23:59:59	1910-01-01T00:00:00	1910-12-31T23:59:59
`../1984`	–	–	1984-01-01T00:00:00	1984-12-31T23:59:59
`vor 1900`	–	–	–	1899-12-31T23:59:59

"Exactly throughout" needs no property of its own: when both pairs name the same seconds, the begin and the end are fixed. The values are xsd:dateTime without a time zone, and they are not UTC or a local zone. A day runs from 00:00:00 to 23:59:59. A time of day after T, including Z, is dropped, so 1985-04-12T23:20:30Z is the day 12 April 1985. That is a rule of this module, not of EDTF.

1900/1910!(1900/1910)=(1900/1910)[1900,1904,=(1908)]

@base <http://example.org/> .
@prefix crm: <http://www.cidoc-crm.org/cidoc-crm/> .
@prefix wtime: <http://wiss-ki.eu/ontology/time/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<event> crm:P4_has_time-span <te> .
<te> a wtime:Time_Expression ;
    crm:P82a_begin_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1910-12-31T23:59:59"^^xsd:dateTime ;
    crm:P86i_contains <te/1> ;
    crm:P1_is_identified_by <te/appellation> .
<te/appellation> a crm:E41_Appellation ;
    crm:P190_has_symbolic_content "1900/1910" .
<te/1> a crm:E52_Time-Span ;
    crm:P82a_begin_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P81a_end_of_the_begin "1900-12-31T23:59:59"^^xsd:dateTime ;
    crm:P81b_begin_of_the_end "1910-01-01T00:00:00"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1910-12-31T23:59:59"^^xsd:dateTime .

<te/1> a crm:E52_Time-Span ;
    crm:P82a_begin_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1910-12-31T23:59:59"^^xsd:dateTime .

The time expression and the appellation are the same as for 1900/1910, with the text "!(1900/1910)".

<te/1> a crm:E52_Time-Span ;
    crm:P82a_begin_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P81a_end_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P81b_begin_of_the_end "1910-12-31T23:59:59"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1910-12-31T23:59:59"^^xsd:dateTime .

<event> crm:P4_has_time-span <te> .
<te> a wtime:Time_Expression ;
    crm:P2_has_type wtime:one_of ;
    crm:P82a_begin_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1908-12-31T23:59:59"^^xsd:dateTime ;
    crm:P86i_contains <te/1>, <te/2>, <te/3> ;
    crm:P1_is_identified_by <te/appellation> .
<te/1> a crm:E52_Time-Span ;
    crm:P82a_begin_of_the_begin "1900-01-01T00:00:00"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1900-12-31T23:59:59"^^xsd:dateTime .
<te/2> a crm:E52_Time-Span ;
    crm:P82a_begin_of_the_begin "1904-01-01T00:00:00"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1904-12-31T23:59:59"^^xsd:dateTime .
<te/3> a crm:E52_Time-Span ;
    crm:P82a_begin_of_the_begin "1908-01-01T00:00:00"^^xsd:dateTime ;
    crm:P81a_end_of_the_begin "1908-01-01T00:00:00"^^xsd:dateTime ;
    crm:P81b_begin_of_the_end "1908-12-31T23:59:59"^^xsd:dateTime ;
    crm:P82b_end_of_the_end "1908-12-31T23:59:59"^^xsd:dateTime .

Each member of a set also gets its own appellation with its own text, such as "=(1908)". A single date does not; its text is on the time expression.

Readings that are not stored¶

?, ~, %, X and significant digits stay in the EDTF text. ~ and % also widen the outer bounds. The module does not write the quoted triples below. They record the reading the text already has:

Typed	Seconds of the outer bounds	Quoted triple
`1900?`	`1900-01-01T00:00:00` – `1900-12-31T23:59:59`	reading `Probable`
`1900~`	`1899-01-01T00:00:00` – `1901-12-31T23:59:59`	reading `Approximate`, and the stated year `1900`
`1900%`	`1899-01-01T00:00:00` – `1901-12-31T23:59:59`	`Probable` and `Approximate`, and the stated year `1900`
`190X`	`1900-01-01T00:00:00` – `1909-12-31T23:59:59`	reading `Unspecified`: exactly one of those years
`1950S2`	`1900-01-01T00:00:00` – `1999-12-31T23:59:59`	reading `Estimated`, and the stated year `1950`

<< <te/1> crm:P82a_begin_of_the_begin "1899-01-01T00:00:00"^^xsd:dateTime >>
    wtime:reading wtime:Approximate ;
    wtime:stated "1900-01-01T00:00:00"^^xsd:dateTime .

Probable is this module's name for EDTF uncertain (?); it does not state a probability. Both bounds of one date carry the same reading. Which component a qualifier covers stays in the model and the text: 2004-06~-11 approximates the year and the month, and the day stays certain. Only outer bounds carry stated. With =(…) the inner bounds keep the stated unit while the outer bounds widen, so the inner bounds already are the stated value and carry the reading alone. A member's reading stays on that member; the envelope has none.

Querying the bounds¶

PREFIX crm: <http://www.cidoc-crm.org/cidoc-crm/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

# Surely going on during 1905.
SELECT ?event WHERE {
  ?event crm:P4_has_time-span/crm:P86i_contains ?span .
  ?span crm:P81a_end_of_the_begin ?surelyFrom ;
        crm:P81b_begin_of_the_end ?surelyUntil .
  FILTER (?surelyFrom <= "1905-01-01T00:00:00"^^xsd:dateTime
       && ?surelyUntil >= "1905-12-31T23:59:59"^^xsd:dateTime)
}

At some time in X: P82a and P82b of a member overlap X.
Surely during X: P81a ≤ start of X and P81b ≥ end of X.
Exactly X: P81a = P82a = start of X and P81b = P82b = end of X.

Use the members, not the envelope, to decide whether a date matches: the envelope of [1900,1904,=(1908)] also spans 1902, which is not a member.

Implementation¶

Class	Role
`Edtf\EdtfSpanParser`	Parses the EDTF profile, `!(…)`, `=(…)`, `vor` / `nach`, into a `TimeSpan` with its model
`Time\TimePeriod`	The model: begin and end points, set members, `oneOf`, the `TimeExtent`, and for an interval the units its begin and end lie in (`latestBegin()`, `earliestEnd()`)
`Time\TimeExtent`	`Edtf`, `Within` (`!`), `Exactly` (`=`)
`Time\TimeBounds`	The four bounds of one date
`Time\TimeGraph`	Turtle for the unit test. It is not written to the store
`Time\TimeModel`	The groups and fields Add the WissKI time model creates, and the uncertainty list on the time-span
`Time\TimeModelInstaller`	Checks the store, inserts the missing classes, generates the paths and saves their bundles and fields as the mapping
`Time\TimeMapping`	The two bundles and the field of each role. Incomplete without both bundles, the dates field and the EDTF text field
`Time\TimeModelWriter`	Writes the type and one time-span per date with its four bounds when the text changed. Nothing without a complete mapping
`Time\SecondRange`	A unit as its first and last second

The typed EDTF string is the source. On save, the sibling integers and the pathbuilder seconds are rewritten from it. The text is not stored a second time. Uncertainty is a separate annotation on the four bounds.

Verbal dates¶

The verbal (MIDAS) field is unchanged. It does not parse EDTF.