View Source

CleverMaps Shell is controlled solely by a set of predefined commands. These commands can be divided in 4 categories, or workflow states. All commands and parameters are case sensitive.

Each command can be further specified by parameters, which can have default values. Each parameter also has a "--" prefix, which is a technicality, and is not mentioned in the tables below, for the sake of readability.

Parameters with string type take string input as a value, if they are mentioned. Parameters with enum type have predefined enumeration of strings, which can be passed as a value. Parameters with no type (-) are passed no values.

Workflow states

Started - you have started the tool
Connected to server - you have successfully logged in to your account on a specific server
Opened project - you have opened a project you have access to
Opened dump - you have created a new dump, or opened an existing one

_

Tool state	Command
Started	`login`
Started	`setup`
Connected to server	`openProject`
	`listProjects`
	`createProject`
	`editProject`
	`deleteProject`
Opened project	`importProject`
	`importDatabase`
	`loadCsv`
	`dumpCsv`
	`dumpProject`
	`listDumps`
	`openDump`
	`truncateProject`
	`validate`
Opened dump	`addMetadata`
	`createMetadata`
	`removeMetadata`
	`renameMetadata`
	`copyMetadata`
	`pushProject`
	`status`
	`applyDiff`
	`diff`

_

Started state

`login`

Log in as a user with correct credentials.

Parameter name	Type	Description	Constraints
`email`	string	your email account	stored in credentials file
`password`	string	your password	stored in credentials file
`server`	string	server to connect to default = `https://secure.clevermaps.io`	stored in config file
`proxyHost`	string	proxy server hostname	stored in config file
`proxyPort`	integer	proxy server port	stored in config file

`setup`

Store your config and credentials in a file so you don't have to specify them each time you log in.

Parameter name	Type	Description	Constraints
`email`	string	your email account	stored in credentials file
`password`	string	your password	stored in credentials file
`server`	string	server to connect to	stored in config file
`dumpDirectory`	string	directory where your dumps will be stored	stored in config file
`proxyHost`	string	proxy server hostname	stored in config file
`proxyPort`	integer	proxy server port	stored in config file

Connected to server

`openProject`

Open a specific project.

Parameter name	Type	Optionality	Description	Constraints
`project`	string		project ID

`listProjects`

List all your projects on the server.

Parameter name	Type	Description	Constraints
`verbose`	-	specifies if the output should be more verbose
`share`	enum	list projects by `share`	`[demo, dimension]`
`organization`	string	list projects by organization (organization ID)

`createProject`

Create a new project.

Parameter name	Type	Description
`title`	string	title of the project
`description`	string	description of the project
`organization`	string	ID of the organization which is the owner of the project

`editProject`

Edit project properties.

Parameter name	Type	Description	Constraints
`project`	string	project ID
`newTitle`	string	new title of the project
`newDescription`	string	new description of the project
`newStatus`	string	new status of the project	`[enabled, disabled]`
`newOrganization`	string	new ID of the organization which is the owner of the project

`deleteProject`

Delete an existing project.

Parameter name	Type	Optionality	Description	Constraints
`project`	string		project ID

Opened project

`importProject`

Allows you to import either:

project from a server (e.g. a data dimension)
create a complete project clone (See the tutorial)
a local dump of a project (See the tutorial)

You can also import a part of a project with one of the parameters (dashboards, datasets, indicators, indicatorDrills, markers, markerSelectors, metrics, views). If you specify none of these parameters, the whole project will be imported. Everytime you specify a datasets parameter, corresponding data will be imported.

Before each import, validate command is called in the background. If there are any model validation violations in the source project, the import is stopped, unless you also provide the --force parameter.

During the import, the origin key is set to all metadata objects. This key indicates the original location of the object (server and the project). This has a special use in case of datasets & data import. import first takes a look at what datasets currently are in the project and compares them with datasets that are about to be imported. Datasets that are not present in the destination project will be imported automatically. In case of datasets that are present in the destination project, 3 cases might occur:

if they have the same name and origin, the dataset will not be imported
if they have the same name but different origin, a warning is shown and the dataset will not be imported
if a prefix is specified, all source datasets will be imported

Parameter name	Type	Description
`server`	string	specifies if the output should be more verbose default = https://secure.clevermaps.io
`project`	string	id of the project to import either `project` or `dump` must be specified
`dump`	string	id of the dump to import either `project` or `dump` must be specified
	string
`prefix`	string	specify a prefix for the metadata objects and data files
`force`	-	force the import even if there are model violations in the source project
`dashboards`	-	import dashboards only
`datasets`	-	import datasets only
`indicators`	-	import indicators only
`indicatorDrills`	-	import indicator drills only
`markers`	-	import markers only
`markerSelectors`	-	import marker selectors only
`metrics`	-	import metrics only
`projectSettings`	-	import project settings only
`shares`	-	import shares only
`views`	-	import views only
`force`	-	ignore source project `validate` errors and proceed with import anyway skip failed dataset dumps (for projects with incomplete data)
`skipData`	boolean	skip data import

Usage examples:

// import all objects referenced from catchment_area_view including datasets & data
importProject --project djrt22megphul1a5 --server --cascadeFrom catchment_area_view

// import all objects referenced from catchment_area_view except including datasets & data
importProject --project djrt22megphul1a5 --server --cascadeFrom catchment_area_view --dashboards --exports --indicatorDrills --indicators --markerSelectors --markers --metrics --views

// import all objects referenced from catchment_area_dashboard
importProject --project djrt22megphul1a5 --server --cascadeFrom catchment_area_dashboard

// import all objects (datasets) referenced from baskets dataset - data model subset
importProject --project djrt22megphul1a5 --server --force --cascadeFrom baskets

`importDatabase`

Allows you to create datasets and import data from an external database.

This command reads the database metadata from which datasets are created, then imports the data and saves them as CSV files. You can choose to import either of which with --skipMetadata and --skipData parameters. Please note that this command does not create any other metadata objects than datasets. It's also possible to import only specific tables using the --tables parameter.

The database must be located on a running database server which is accessible under an URL. This can be on localhost, or anywhere on the internet. Valid credentials to the database are of course necessary.

So far, the command supports these database engines:

PostgreSQL (https://www.postgresql.org/)

Parameter name	Type	Description	Constraints
`engine`	enum	name of the database engine	[postgresql]
`host`	string	database server hostname for local databases, use `localhost`
`port`	integer	database server port
`schema`	string	name of the database schema leave out if your engine does not support schemas, or the schema is `public`
`database`	string	name of the database
`user`	string	user name for login to the database
`password`	string	user's password
`tables`	array	list of tables to import leave out if you want to import all tables from the database example = `"orders,clients,stores"`
`skipData`	boolean	skip data import
`skipMetadata`	boolean	skip metadata import

Usage examples:

importDatabase --engine postgresql --host localhost --port 5432 --database my_db --user postgres --password test
importDatabase --engine postgresql --host 172.16.254.1 --port 6543 --schema my_schema --database my_db --user postgres --password test --tables orders,clients,stores

`loadCsv`

Load data from a CSV file into a specified dataset.

loadCsv also offers various CSV input settings. Your CSV file may contain specific features, like custom quote or separator characters. The parameters with the csv prefix allow you to configure the data load to fit these features, instead of transforming the CSV file to one specific format. Special cases include the csvNull and csvForceNull parameters.

csvNull allows you to specify a value, which will be interpreted as a null value
- e.g. "false" or "_"
csvForceNull then specifies on which columns the custom null replacement should be enforced
- e.g. "name,title,description"

Parameter name	Type	Description	Constraints
`file`	string	path to the CSV file one of `file`, `s3Uri` or `url` parameters must be specified
`s3Uri`	string	URI of an object on AWS S3 to upload (see examples below) one of `file`, `s3Uri` or `url` parameters must be specified
`url`	string	HTTPS URL which contains a CSV file to be loaded into the dataset one of `file`, `s3Uri` or `url` parameters must be specified
`dataset`	string	dataset to into which the data should be loaded
`mode`	enum	`incremental` mode appends the data to the end of the table `full` mode truncates the table and loads the table anew	`[incremental, full]`
`csvHeader`	boolean	specifies if the CSV file to upload has a header default = `true`	`[true, false]`
`csvSeparator`	char	specifies the CSV column separator character default = `,`
`csvQuote`	char	specifies the CSV quote character default = `"`
`csvEscape`	char	specifies the CSV escape character default = `\`
`csvNull`	string	specifies the replacement of custom CSV null values
`csvForceNull`	enum	specifies which CSV columns should enforce the null replacement
`verbose`	boolean	enables more verbose output default = `false`	`[true, false]`
`multipart`	boolean	enables multipart file upload (recommended for files larger than 2 GB) default = true	`[true, false]`
`gzip`	boolean	enables gzip compression default = true	`[true, false]`

Usage examples:

Please note that your AWS S3 Access Key ID and Secret Access Key must be set using setup command first.

loadCsv --dataset orders --mode full --s3Uri s3://my-company/data/orders.csv --verbose

loadCsv --dataset orders --mode full --url http://www.example.com/download/orders.csv --verbose

`dumpCsv`

Dump data from a specified dataset into a CSV file.

Parameter name	Type	Optionality	Description	Constraints
`file`	string		path to the CSV file	stored in config file
`dataset`	string		dataset whose data will be dumped

`dumpProject`

Dump project data & metadata to a directory. If the dump if successfull, the current dump is opened.

Parameter name	Type	Description	Constraints
`directory`	string	directory to which the dump will be saved	stored in config file
`skipMetadata`	-	skip metadata dump
`skipData`	-	skip data dump
`force`	-	skip failed dataset dumps (for projects with incomplete data)

`listDumps`

List all local project dumps.

Parameter name	Type	Optionality	Description	Constraints
`directory`	string		parent directory of a specific project's dumps	stored in config file

`openDump`

Open a specific dump.

Parameter name Type Optionality Description Constraints

directory

string

parent directory of a specific project's dumps

stored in config file

dump

string

open a specific dump

if not specified, latest dump is opened

`truncateProject`

Deletes all metadata and data from the project.

This command has no parameters.

`validate`

Validate the project's data model and data.

Parameter name	Type	Description
`project`	string	validate any other project than the currently opened one
`skipModel`	string	skip validations of the data model
`skipData`	string	skip validations of the data itself

Opened dump

`addMetadata`

Add a metadata object to the project. The file must be located in a currently opened dump, and in the correct directory.

If the --objectName parameter is not specified, addMetadata will add all new objects in the current dump.

Parameter name	Type	Optionality	Description	Constraints
`objectName`	string		name of the object (with or without .json extension)

`createMetadata`

Create a new metadata object.

At this moment, only dataset type is supported. Datasets are generated from a provided CSV file.

Parameter name	Type	Description	Constraints
`type`	enum	type of the object	`[dataset]`
`objectName`	string	current name of the object (with or without .json extension)
`subtype`	enum	name of the object copy (with or without .json extension) required only for `dataset` type	`[basic, geometryPoint, geometryPolygon]`
`file`	string	path to the CSV file (located either in dump, or anywhere in the file system) required only for `dataset` type
`primaryKey`	string	name of the CSV column that will be marked as primary key if not specified, first CSV column is selected
`geometry`	string	name of the `geometry` key required only for `geometryPolygon` subtype
`csvSeparator`	char	specifies custom CSV column separator character default = `,`
`csvQuote`	char	specifies custom CSV column quote character default = `"`
`csvEscape`	char	specifies custom CSV column escape character default = `\`

Usage examples:

createMetadata --type dataset --subtype basic --objectName "baskets" --file "baskets.csv" --primaryKey "basket_id"
createMetadata --type dataset --subtype geometryPoint --objectName "shops" --file "shops.csv" --primaryKey "shop_id"
createMetadata --type dataset --subtype geometryPolygon --objectName "district" --geometry "districtgeojson" --file "district.csv" --primaryKey "district_code"

`removeMetadata`

Remove a metadata object from the project and from the dump. The file must be located in a currently opened dump, and must not be new.

Parameter name Type Optionality Description Constraints

objectName

string

name of the object (with or without .json extension)

one of objectName, objectId or orphanObjects parameters must be specified

objectId string id of the object

orphanObjects

boolean

prints a list of removeMetadata commands to delete orphan metadata objects

orphan object is an object not referenced from any of the project's views, or visible anywhere else in the app

`renameMetadata`

Rename an object in a local dump and on the server. If the object is referenced in some other objects by URI (/rest/projects/$projectId/md/{objectType}?name=), the references will be renamed as well.

Parameter name	Type	Optionality	Description	Constraints
`objectName`	string		current name of the object (with or without .json extension)
`newName`	string		new name of the object (with or without .json extension)

`copyMetadata`

Create a copy of an object existing in a currently opened dump.

This command unwraps the object from a wrapper, renames it and removes generated common syntax keys. If the objectName and newName arguments are the same, the object is unwrapped only.

Parameter name	Type	Optionality	Description	Constraints
`objectName`	string		current name of the object (with or without .json extension)
`newName`	string		name of the object copy (with or without .json extension)

`pushProject`

Upload all modified files (data & metadata) to the project.

This command basically wraps the functionality of loadCsv. It collects all modified metadata to upload, and performs full load of CSV files.

Parameter name	Type	Description	Constraints
`skipMetadata`	-	skip metadata push
`skipData`	-	skip data push
`skipValidate`	-	skip the run of `validate` after push
`force`	boolean	force metadata push when there's a share breaking change
`verbose`	boolean	enables more verbose output default = `false`	`[true, false]`
`multipart`	boolean	enables multipart file upload (recommended for files larger than 2 GB) default = true	`[true, false]`
`gzip`	boolean	enables gzip compression default = true	`[true, false]`

`status`

Check the status of a currently opened dump against the project on the server.

This command detects files which have been locally or remotely changed, and also detects files which have a syntax error or constraint violations.

Parameter name	Type	Optionality	Description	Constraints
`remote`	-		list metadata contents on the server

`applyDiff`

Create and apply metadata diff between two live projects.

This command compares all metadata objects of the currently opened project and project specified in the --sourceProject command, and applies changes to the currently opened dump. Metadata objects in dump can be either:

added (completely new objects that aren't present in currently opened project)
modified
deleted (deleted objects not present in the sourceProject)

When the command finishes, you can review the changes applied to the dump using either status or diff comands. The command then tells you to perform specific subsequent steps. This can be one of (or all) these commands:

addMetadata (to add the new objects)
pushProject (to push the changes in modified objects)
removeMetadata (to remove the deleted objects - a command list which must be copypasted to Shell is generated )

Parameter name Type Optionality Description Constraints

sourceProject

string

id of the source project

objectTypes

string

list of object types to be compared

example = "views,indicators,metrics"

`diff`

Compare local metadata objects with those in project line by line.

If the --objectName parameter is not specified, all wrapped modifed objects are compared.

Parameter name	Type	Optionality	Description	Constraints
`objectName`	string		name of a single object to compare (with or without .json extension)

The command outputs sets of changes - "deltas" done in each object. Each object can have multiple deltas, which are followed by header with syntax:

/{objectType}/{objectName}.json
[ A1 A2 | B1 B2 ]
...

Where:
A1 = start delta line number in the dump file
A2 = end delta line number in the dump file
B1 = start delta line number in the remote file
B2 = end delta line number in the remote file

Specific output example:

CleverMaps for Developers > Command list > diff-2.png

Which means lines 82-85 in the dump object have been added (+) in favor of lines 82-85, which have been removed (−) from the remote object.