Deployment

Prerequisites

The application consists of two repositories: a Spark-powered Java backend and a Javascript frontend written with React and Redux. To install and deploy the application, you will need Java 8 and Maven for the datatools-server and Node (>= v10 required, >= v14 recommended), npm, yarn and mastarm for the datatools-ui.

User authentication is done via Auth0. You will need an Auth0 account and application to use the Data Manager.

Two databases are required for the application:

  • MongoDB (>=v2.6.4) is used to store application-level data (e.g., persisting projects and feed sources) and
  • PostgreSQL (>=v9.6) for use by gtfs-lib to store GTFS data.

Installation and Basic Configuration

Clone the repo and change to the project directory:

$ git clone https://github.com/ibi-group/datatools-ui.git
$ git clone https://github.com/ibi-group/datatools-server.git

Copy the included configuration env templates for both the server and UI repos:

$ cp datatools-ui/configurations/default/env.yml.tmp datatools-ui/configurations/default/env.yml
$ cp datatools-server/configurations/default/env.yml.tmp datatools-server/configurations/default/env.yml

You'll then need to supply Auth0 account information (see below) and API keys for various services used in the application.

The default server.yml for datatools-server and settings.yml for datatools-ui should work out of the box, but you may want to specify alternative settings files outside of these repositories. These can be specified as a directory during datatools-ui build with mastarm:

$ mastarm build --config /path/to/configurations/dir

AND as individual file args for datatools-server:

$ java -jar target/dt-v1.0.0.jar /path/to/env.yml /path/to/server.yml

In datatools-server:server.yml, be sure to update the paths for the place where databases will be stored:

application:
  data:
    gtfs: /path/to/gtfs/storage

Database setup

GTFS data storage

GTFS data storage is handled by whichever standard RDBMS you prefer. However, the application has been significantly tuned and optimized for PostgreSQL 9, so we highly recommend using PostgreSQL.

Once PostgreSQL is installed and the service has been started, create the database for instance by using the CLI createdb:

$ createdb gtfs_storage_db

Pass the URL of the database in the datatools-server's env.yml (and optionally add additional connection variables):

GTFS_DATABASE_URL: jdbc:postgresql://localhost/gtfs_storage_db
# GTFS_DATABASE_USER:
# GTFS_DATABASE_PASSWORD:

Application data storage

Application data storage (i.e., where projects, feed sources, and feed versions are stored) is handled by MongoDB. There is no need to manually initialize a database in MongoDB (MongoDB will handle this automatically if you prefer). Connection details for MongoDB are also set in the datatools-server's env.yml:

MONGO_URI: # defaults to localhost:27017 (MongoDB default) if empty
MONGO_DB_NAME: application_db

Setting up Auth0

Auth0 is used for authentication in the application. If you don't need authentication (e.g., if you're running locally or on a secured network), you can set DISABLE_AUTH to true for both the server and UI configurations (env.yml).

Creating account and application (client)

  1. Create an Auth0 account (free).
  2. Once you've created an Auth0 account, create an application (client) in Auth0 to use with the Data Manager with the following settings:
    • enable only Username-Password-Authentication connections (i.e., turn off Google)
    • set Use Auth0 instead of the IdP to do Single Sign On to true
    • update the following application- and account-level settings to include http://localhost:9000 (or the domain where the application will be hosted):
      • Account level (Account Settings > Advanced)
        • Allowed logout URLs
      • Application level
        • Allowed Callback URLs
        • Allowed Origins (CORS)
    • Keep all other default settings

Creating your first user

Create your first Auth0 user through Auth0 web console (Users > Create User). In addition to an email and password, you'll need to supply the user with the following default application admin app_metadata (user_metadata should remain empty):

{
  "datatools": [{
    "permissions": [
      {
        "type": "administer-application"
      }
    ],
    "subscriptions": [],
    "projects": [],
    "client_id": "your-auth0-client-id"
  }]
}

Update env.yml for server and UI repos

Update the following properties in both env.yml files to reflect the public Auth0 application settings:

AUTH0_DOMAIN: your-auth0-domain.auth.com
AUTH0_CLIENT_ID: your-auth0-client-id

Update the following properties in datatools-server and env.yml to reflect the secure Auth0 application settings.

Note: For older Auth0 accounts or tenants, it is possible to use the Auth0 secret token with the HS256 algorithm is possible. However, newer Auth0 tenants will need to specify the absolute path of their .pem file in the AUTH0_PUBLIC_KEY property. This public key only needs to be downloaded one time for your Auth0 tenant at https://[your_domain].auth0.com/pem.

AUTH0_SECRET: your-auth0-client-secret # used for pre-September 2017 Auth0 accounts
AUTH0_PUBLIC_KEY: /location/of/auth0-account.pem # used for post-September 2017 Auth0 accounts
AUTH0_TOKEN: your-auth0-api-token

Note: To generate the api_token, go to Documentation > Management API. After adding scopes, your token will appear in the input field.

To allow for the creation, deletion and editing of users you must generate a token for the following scopes:

  • users:
    • read, update, create and delete
  • users_app_metadata:
    • read, update, create and delete

Auth0 Post-Login Action Configuration: making app_metadata and user_metadata visible via token

If using OIDC-conformant clients/APIs (which appears to be mandatory for new Auth0 tenants), you must set up a custom Auth0 action to add app_metadata and user_metadata to the user's id token (Note: this is not the default for older, "legacy" Auth0 accounts).

To set up the action, go to Actions > Flows > Login, then under Add action > Custom, click Create Action. Fill in the action name and pick a recommended runtime, and click Create. Modify the function onExecutePostLogin as follows, then click Save Draft:

exports.onExecutePostLogin = async (event, api) => {
  if (event.authorization) {
    const namespace = 'http://datatools';
    api.idToken.setCustomClaim(`${namespace}/user_metadata`, event.user.user_metadata);
    api.idToken.setCustomClaim(`${namespace}/app_metadata`, event.user.app_metadata);
  }
};

If you want the rule to apply only to specific clients, you can retain the conditional block that checks the context.clientID value. Otherwise, you can remove this conditional block if it's not needed. This rule will ensure that app_metadata and user_metadata are included in the user's token, as required for OIDC-conformant clients/APIs in new Auth0 tenants.

You can test the action with mock token data using the Test tab. Once ready, click Deploy, then click Back to Flow. In the diagram, drag the action between the Start and Complete steps, then click Apply. You can test that the action is correctly executed by logging-in to datatools with an admin user and checking that the Admin functionality is available.

Building and Running the Application

Install the Javascript dependencies for datatools-ui using yarn:

$ cd datatools-ui
$ yarn

Build and deploy datatools-ui to s3 using npm script (which calls mastarm):

$ npm run deploy -- s3://$S3_BUCKET_NAME/dist

Package datatools-server using Maven:

$ cd datatools-server
$ mvn package

Deploy datatools-server with Java:

$ java -jar target/dt-v1.0.0.jar /path/to/env.yml /path/to/server.yml

Datatools-server should now be running at http://localhost:9000 (or whatever port you specified in server.yml). Datatools-ui assets are obtained from the dist folder relative the url specified in server.yml at application.client_assets_url. While running a development server of datatools-ui, these assets are delivered to the client using budo, so the links defined in the backend server.yml are only used in a production setting.

Configuring Modules

The application contains a number of optional modules that each require their own configuration settings and resources. At minimum, each module must be set to enabled: true and may require additional configuration.

Note: for datatools-server v3.0.0, the editor and r5_network should be disabled because they have not been refactored to handle updates to the loading of GTFS data into an RDBMS. Please use v2.0.0 or wait for releases following v3.0.0.

Editor

Enables the GTFS Editor module.

List of configuration settings

  • MAPZEN_TURN_BY_TURN_KEY
  • MAPBOX_MAP_ID
  • MAPBOX_ACCESS_TOKEN
  • R5_URL (optional parameter for r5 routing in editor pattern drawing)

R5 network validation

While the application handles basic validation even without the r5_network module enabled, this module allows for enhanced accessibility- and map-based validation.

List of configuration settings

  • OSM_VEX - datatools-server:env.yml the validator requires the URL of a running instance of the OSM vex server.

Alerts

Enables the real-time alerts module.

List of configuration settings

  • use_extension - extension key to use for alerts
  • url
  • Requires gtfsapi module

GTFS+ (gtfsplus)

Enables the GTFS+ module.

Deployment

Enables the OTP automated deployments module.

List of configuration settings

  • OSM_VEX - datatools-server:env.yml the validator requires the URL of a running instance of the OSM vex server.

GTFS API

Supports other modules with API endpoints for getting GTFS entities (routes, stops, etc.)

List of configuration settings

  • load_on_fetch - whether to load GTFS feeds when new feed is detected
  • load_on_startup - whether to load GTFS feeds on application startup
  • use_extension - which extension to connect to
  • update_frequency - update frequency for GTFS API (in seconds). Comment out to disable updates.

Configuring Extensions

The application supports integration with several third-party resources for retrieving feeds.

Integration with transit.land

Ensure that the extensions:transitland:enabled flag is set to true in config.yml:

extensions:
  transitland:
    enabled: true
    api: https://transit.land/api/v1/feeds

Integration with TransitFeeds

Note: TransitFeeds is not regularly updated and is being replaced by the MobilityData Database

Ensure that the extensions:transitfeeds:enabled flag is set to true in config.yml, and provide your API key:

extensions:
  transitfeeds:
    enabled: true
    api: http://api.transitfeeds.com/v1/getFeeds
    key: your-api-key