SCHEDULE: Difference between revisions

From Open Rail Data Wiki
Updated information about how to download files, added detail to interpretation of schedule files
Line 1: Line 1:
= Schedule data =
The SCHEDULE feed is an extract of train schedules from Network Rail's ITPS (Integrated Train Planning System), converted in to JSON format.  Network Rail are not planning to make raw CIF files available.


== Overview ==
Schedules can either be downloaded for all TOCs, or for individual TOCs.  Two types of schedule file are available - a 'full' file which contains a snapshot of all schedules at that point in time, and an 'update' file which can be applied to an older set of files to bring them up-to-date.


The Schedule feed is an extract of train schedules from Network Rail's ITPS (Integrated Train Planning System), converted in to JSON format for easier parsing.  Network Rail are not planning to make raw CIF files available.
The [http://www.atoc.org/about-atoc/rail-settlement-plan/data-feeds/types-of-data CIF User Specification] is available from ATOC's website, which details the format of the CIF file and the fields within, and should be read in conjunction with these wiki pages.


Schedule files are available for all passenger TOCs and for each TOC.  Two types of file are available - a 'full' file which contains a snapshot of all schedules, and an 'update' file which can be applied to a a local database to bring it up-to-date with any changes.
= Downloading =


The [http://www.atoc.org/about-atoc/rail-settlement-plan/data-feeds/types-of-data CIF User Specification] is available from ATOC's website, which details the format of the CIF fileThis will be useful to developers wishing to gain deep understanding about the way train scheduling works, above and beyond the information contained here.
The schedule data is compressed using gzip and is downloaded from Amazon S3 via a private URL which is valid for a few minutes after generation.  Authentication will be required either by logging in through the datafeeds website, or by sending your username and password as part of the HTTP requestOn successful authentication, you will receive a 403 redirect to the location of the schedule files. Requests are of the format:


== Downloading ==
  https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=type&day=day


The schedule data, compressed using gzip, is downloaded from Amazon S3 via a private URL which is valid for a few minutes after generation.
where ''type'' determines the TOCs contained in the schedule extract and whether the file is a full schedule or an update file, and ''day'' determines which day's update file is downloaded. For example:


To request schedule data, send an HTTP request with your username and password to:
   https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full
 
   https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=bucket&day=file
 
For example:


  https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full
would download a full schedule file for all TOCs.


'type' and 'day' values should be one of the following:
Values for ''type'' and ''day'' can be one of the following:


{| class='wikitable'
{| class='wikitable'
!Type
!''type''
!Day
!''day''
!Description
!Description
!Example URL
!Example URL
Line 35: Line 31:
|-
|-
|CIF_ALL_UPDATE_DAILY
|CIF_ALL_UPDATE_DAILY
|toc-update-DAY
|toc-update-''DAY''
|Daily all-TOC update
|Daily all-TOC update
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sun Sun],  
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sun Sun],  
|-
|-
|CIF_XX_TOC_FULL_DAILY
|CIF_''XX''_TOC_FULL_DAILY
|toc-full
|toc-full
|Daily snapshot for TOC XX
|Daily snapshot for TOC ''XX''
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_FULL_DAILY&day=toc-full Daily]
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_FULL_DAILY&day=toc-full Daily]
|-
|-
|CIF_XX_TOC_UPDATE_DAILY
|CIF_''XX''_TOC_UPDATE_DAILY
|toc-update-DAY
|toc-update-''DAY''
|Daily update for TOC XX
|Daily update for TOC ''XX''
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sun Sun]
|[https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-mon Mon], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-Tue Tue], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-wed Wed], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-thu Thu], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-fri Fri], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sat Sat], [https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_XX_TOC_UPDATE_DAILY&day=toc-update-sun Sun]
|-
|-
|}
|}


'''DAY''' should be replaced with the date of the previous day.  For example, on Monday, Sunday's ('sun') update should be requested.
''DAY'' should be replaced with the shortened version of the name of the previous day.  For example, on Monday, Sunday's ('sun') update should be requested.


A list of [[TOC Codes]] is available.  Note that the TOC code used in the URL is not the ATOC code as seen in schedule files, but the business code as used in Network Rail systems.
A list of [[TOC Codes]] is available.  Note that the TOC code used in the URL is not the ATOC code as seen in schedule files, but the business code as used in Network Rail systems.


Replace '''bucket''' with the name of the bucket, and '''file''' with the name of the file.  On successful authentication, you will receive a 403 redirect to the location of the schedule files.


=== Downloading via curl ===
== Downloading via curl ==


The schedule file can be downloaded from the command line using the ''curl'' utility (included with linux, versions available for other operating systems) with the following syntax:
The schedule file can be downloaded from the command line using the ''curl'' utility (included with linux, versions available for other operating systems) with the following syntax:


<code>
  curl -L -u username:password -o file.gz 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full'
curl -L -u username:password -o file.gz 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full'
</code>


''Thanks to Jules for working this syntax out''
(Thanks to Jules for figuring this syntax out.)


== Data ==
= File Format =


The schedule data contains a header row, a set of zero or more association records, a set of zero or more schedule records, and an end-of-file (EOF) record.
Schedule data is contained in the schedule file in JSON format, with a single JSON record per line.  The file can be parsed line-by-line to extract the data required.  Each schedule file contains:


Each association and schedule record has an action - either 'create' or 'delete'.  In full files, there will be no 'delete' records.
* A header row
* Zero or more association records
* Zero or more schedule records
* and an end-of-file (EOF) record.


Update files must be applied sequentially to a full file.
Each association and schedule record has an action - either 'create' or 'delete'.  These are used in update files to add or remove records from the master schedule.  In full files, there will be no 'delete' records.  Update files must be applied sequentially to a full file.


== Interpretation ==
== Association Records ==


=== Validity ===
Associations are relationships between two schedules (i.e. a main train and an associated train.)  There are three types of association:


Associations and schedule validities are between a start date and an end date, and on particular days of the weekThey each have a Short Term Planning (STP) indicator field as follows:
* '''NP''' - Next Train.  Not present for all schedules, but indicates the UID of the next service that the vehicle on this service will work
* '''JJ''' - Join.  Occurs at the end of the associated train's schedule.
* '''VV''' - SplitOccurs at an intermediate location of the main train's schedule and indicates another train services that part of this train will form.


* '''C''' - Planned cancellation: the schedule does not apply on this date, and the train will not run.  Typically seen on public holidays when an alternate schedule applies, or on Christmas Day.
Associations may be for the same day ('''S'''), or cross midnight either backward ('''P''') or forward ('''N''') depending on the date indicator field.
* '''N''' - STP schedule: similar to a permanent schedule, but planned through the Short Term Planning process
* '''O''' - Overlay schedule: an alteration to a permanent schedule
* '''P''' - Permanent schedule: a schedule planned through the Long Term Planning process
 
For any date a permanent schedule ('P') is valid, it may be overridden by an overlay ('O') or a planned cancellation ('C'). However, if any two schedules appear to be valid for a particular day, the schedule with the lowest alphabetical STP indicator wins.


=== Schedules ===
== Schedule Records ==


A schedule comprises a header containing a schedule UID, data about the schedule (including whether it is a train, bus or ship) and validity dates, and an ordered list of locations and times at which a particular service should arrive, depart or pass.
A schedule record comprises a header containing a schedule UID, data about the schedule (including whether it is a train, bus or ship) and validity dates, and an ordered list of locations and times at which a particular service should arrive, depart or pass.


* Originating locations will always have a WTT departure time and optionally a public departure time
* Originating locations will always have a WTT departure time and optionally a public departure time
Line 98: Line 91:
* A location may have engineering, pathing or performance allowances
* A location may have engineering, pathing or performance allowances


=== Associations ===
== Validity ==


Associations are relationships between two schedules - a main train and an associated train.
Associations and schedule validities are between a start date and an end date, and on particular days of the week. They each have a Short Term Planning (STP) indicator field as follows:


There are three types of association:
* '''C''' - Planned cancellation: the schedule does not apply on this date, and the train will not run.  Typically seen on public holidays when an alternate schedule applies, or on Christmas Day.
* '''N''' - STP schedule: similar to a permanent schedule, but planned through the Short Term Planning process
* '''O''' - Overlay schedule: an alteration to a permanent schedule
* '''P''' - Permanent schedule: a schedule planned through the Long Term Planning process


* '''NP''' - Next Train.  Not present for all schedules, but indicates the UID of the next service that the vehicle on this service will work
For any date a permanent schedule ('P') is valid, it may be overridden by an overlay ('O') or a planned cancellation ('C'). However, if any two schedules appear to be valid for a particular day, the schedule with the lowest alphabetical STP indicator wins.
* '''JJ''' - Join.  Occurs at the end of the associated train's schedule.
* '''VV''' - Split.  Occurs at an intermediate location of the main train's schedule and indicates another train services that part of this train will form.
 
Associations may be for the same day ('''S'''), or cross midnight either backward ('''P''') or forward ('''N''') depending on the date indicator field.

Revision as of 01:58, 11 May 2013

The SCHEDULE feed is an extract of train schedules from Network Rail's ITPS (Integrated Train Planning System), converted in to JSON format. Network Rail are not planning to make raw CIF files available.

Schedules can either be downloaded for all TOCs, or for individual TOCs. Two types of schedule file are available - a 'full' file which contains a snapshot of all schedules at that point in time, and an 'update' file which can be applied to an older set of files to bring them up-to-date.

The CIF User Specification is available from ATOC's website, which details the format of the CIF file and the fields within, and should be read in conjunction with these wiki pages.

Downloading

The schedule data is compressed using gzip and is downloaded from Amazon S3 via a private URL which is valid for a few minutes after generation. Authentication will be required either by logging in through the datafeeds website, or by sending your username and password as part of the HTTP request. On successful authentication, you will receive a 403 redirect to the location of the schedule files. Requests are of the format:

 https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=type&day=day

where type determines the TOCs contained in the schedule extract and whether the file is a full schedule or an update file, and day determines which day's update file is downloaded. For example:

 https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full

would download a full schedule file for all TOCs.

Values for type and day can be one of the following:

type day Description Example URL
CIF_ALL_FULL_DAILY toc-full Daily all-TOC snapshot Daily
CIF_ALL_UPDATE_DAILY toc-update-DAY Daily all-TOC update Mon, Tue, Wed, Thu, Fri, Sat, Sun,
CIF_XX_TOC_FULL_DAILY toc-full Daily snapshot for TOC XX Daily
CIF_XX_TOC_UPDATE_DAILY toc-update-DAY Daily update for TOC XX Mon, Tue, Wed, Thu, Fri, Sat, Sun

DAY should be replaced with the shortened version of the name of the previous day. For example, on Monday, Sunday's ('sun') update should be requested.

A list of TOC Codes is available. Note that the TOC code used in the URL is not the ATOC code as seen in schedule files, but the business code as used in Network Rail systems.


Downloading via curl

The schedule file can be downloaded from the command line using the curl utility (included with linux, versions available for other operating systems) with the following syntax:

 curl -L -u username:password -o file.gz 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full'

(Thanks to Jules for figuring this syntax out.)

File Format

Schedule data is contained in the schedule file in JSON format, with a single JSON record per line. The file can be parsed line-by-line to extract the data required. Each schedule file contains:

  • A header row
  • Zero or more association records
  • Zero or more schedule records
  • and an end-of-file (EOF) record.

Each association and schedule record has an action - either 'create' or 'delete'. These are used in update files to add or remove records from the master schedule. In full files, there will be no 'delete' records. Update files must be applied sequentially to a full file.

Association Records

Associations are relationships between two schedules (i.e. a main train and an associated train.) There are three types of association:

  • NP - Next Train. Not present for all schedules, but indicates the UID of the next service that the vehicle on this service will work
  • JJ - Join. Occurs at the end of the associated train's schedule.
  • VV - Split. Occurs at an intermediate location of the main train's schedule and indicates another train services that part of this train will form.

Associations may be for the same day (S), or cross midnight either backward (P) or forward (N) depending on the date indicator field.

Schedule Records

A schedule record comprises a header containing a schedule UID, data about the schedule (including whether it is a train, bus or ship) and validity dates, and an ordered list of locations and times at which a particular service should arrive, depart or pass.

  • Originating locations will always have a WTT departure time and optionally a public departure time
  • Intermediate locations in a schedule will have a passing time if they are a mandatory timing point, or an arrival and departure time if the train carries out an activity at that location
  • Terminating locations will always have a WTT arrival time and optionally a public arrival time, which may be some minutes later than the WTT time
  • A location may have one or more activities associated with it - for example, U for locations where the train calls to pick up passengers (i.e. not available for alighting), D for locations where the train calls to set down passengers (i.e. not available for boarding).
  • A location may have engineering, pathing or performance allowances

Validity

Associations and schedule validities are between a start date and an end date, and on particular days of the week. They each have a Short Term Planning (STP) indicator field as follows:

  • C - Planned cancellation: the schedule does not apply on this date, and the train will not run. Typically seen on public holidays when an alternate schedule applies, or on Christmas Day.
  • N - STP schedule: similar to a permanent schedule, but planned through the Short Term Planning process
  • O - Overlay schedule: an alteration to a permanent schedule
  • P - Permanent schedule: a schedule planned through the Long Term Planning process

For any date a permanent schedule ('P') is valid, it may be overridden by an overlay ('O') or a planned cancellation ('C'). However, if any two schedules appear to be valid for a particular day, the schedule with the lowest alphabetical STP indicator wins.