This tutorial describes advanced project management techniques in CleverMaps Shell.
In the first tutorial, you've learned how to create a project, dump it, import a dimension and add some data & metadata. Now the project has been handed over and your clients are using it. So what about extensions? What about data updates? We sure don't want our clients to use an incomplete/broken project. We'll create a clone, perform our work there, and then elegantly put our changes back to the live project. Here's how.
Creating a project clone
The process of creating a project clone is very simple.
Log in to CleverMaps Shell and use cloneProject
command.
CleverMaps$ login Found config file: /home/tomas/.cleverMaps/config * setting https://secure.clevermaps.io as default server * setting /var/local/metadata as default dump directory Successfully logged in as tomas.schmidl@clevermaps.io (using access token) tomas.schmidl@secure.clevermaps.io$ cloneProject --project k5t8mf2a80tay2ng --organization <your_organization_id> Created project c2ovon61uudj9lpm Project c2ovon61uudj9lpm successfully opened Project title: Clone of First project Organization: <your_organization> Opening dump... You don't have dump to open for this project. Starting cloning of project k5t8mf2a80tay2ng... Cloning of project m4slgmqh2ihbm62k was successful. tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm$
Cloning of project can take a few minutes. If you are having trouble with cloning, you can try server side import method below.
<your_organization_id> is provided with your CleverMaps license
After successfully cloning project, use dumpProject to dump your project.
This creates an identical clone of the original project. A clone might serve multiple purposes. It can be used as:
- a working copy of the original project (where updates are made, and which later replaces the original project)
- a base for a branched out, similar project (in order not to start from scratch)
- a backup of older version of the project (to be archived)
Restoring a backup dump
Each dump is an image of a project in a specific time. So, a local dump can be used as a backup of a project. It is useful to periodically dump the project you're working on, to save your previous work. This way, in case you break the live project, you can always revert to a previous version of the project.
The first step is to truncate the live project of all data and metadata. Open the project again to close the curent dump, and use the truncateProject
command.
tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng/dump:2019-01-30_14-46-52$ openProject --project c2ovon61uudj9lpm Project c2ovon61uudj9lpm (First project (clone)) successfully opened tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm$ truncateProject Truncation of project started... Project was successfully truncated. tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm$
Then use listDumps
to list previous dumps of the project.
tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm$ listDumps Listing dumps of project c2ovon61uudj9lpm... 2019-01-30_14-08-31 Dump date: Wed Jan 30 14:08:31 CET 2019 Dump server: https://secure.clevermaps.io Project title: First project (clone) Data content: orp_dwh.csv (24 KB) cast_obec_dwh.csv (1 MB) zsj_d_dwh.csv (3 MB) obec_dwh.csv (761 KB) okres_dwh.csv (9 KB) transactions.csv (1 MB) zsj_dwh.csv (2 MB) obec_momc_dwh.csv (754 KB) kraj_dwh.csv (1 KB) Metadata content: 1 metrics, 17 datasets, 1 indicators, 1 dashboards, 1 views, 1 indicatorDrills 2019-01-30_14-46-52 Dump date: Wed Jan 30 14:46:52 CET 2019 Dump server: https://secure.clevermaps.io Project title: First project (clone) Data content: orp_dwh.csv (24 KB) cast_obec_dwh.csv (1 MB) zsj_d_dwh.csv (3 MB) obec_dwh.csv (761 KB) okres_dwh.csv (9 KB) transactions.csv (1 MB) zsj_dwh.csv (2 MB) obec_momc_dwh.csv (754 KB) kraj_dwh.csv (1 KB) Metadata content: 1 metrics, 17 datasets, 1 indicators, 1 dashboards, 1 views, 1 indicatorDrills tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm$
We've decided that we want to roll the project back to the previous dump with id 2019-01-30_14-08-31
. This is possible by specifying the --dump
parameter.
tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm$ importProject --dump 2019-01-30_14-08-31 Checking if the project c2ovon61uudj9lpm is empty... OK Importing local dump 2019-01-30_14-08-31 of project c2ovon61uudj9lpm...
Import with --dump
parameter will first check if the project is empty. It is, because we've truncated it earlier. This behaviour can be overriden by specifying the --force
parameter.
Let's skip some of the importProject
command output again...
Import finished! Source project: c2ovon61uudj9lpm (First project (clone)) Source dump: 2019-01-30_14-08-31 Destination project: c2ovon61uudj9lpm (First project (clone)) Destination dump: 2019-01-30_16-57-02 Imported metadata objects: 22 Imported CSV files: 9 To view detailed imported content, use status command. When you're done, use addMetadata command to add the metadata to the project. To push the data to the project, use pushProject command. tomas.schmidl@secure.clevermaps.io/project:c2ovon61uudj9lpm/dump:2019-01-30_16-57-02$
The final step to restore the backup dump is to upload the data and metadata to the project. As usually, use addMetadata
and pushProject
to do that.
Import usage
The importProject
command can import either a project from a server, or a local dump. What to import is decided based on provided parameters.
importProject --project {projectId} --server {serverName}
- imports a project from a server (which defaults to https://secure.clevermaps.io)
importProject --project {projectId} --dump {dumpId}
- imports a local dump of another project (not the one currently opened)
importProject --dump {dumpId}
- imports a dump of currently opened project
Updating a live project
Now that we've done some work on the project clone, we want to transfer the changes to the live project.
So with the premise that we don't want our clients to use an incomplete/broken project still in mind, what can we do? Before we update the data and metadata, we will set the project's status to disabled
, using the editProject
command.
tomas.schmidl@secure.clevermaps.io$ editProject --project k5t8mf2a80tay2ng --newStatus disabled Project status changed to "disabled" tomas.schmidl@secure.clevermaps.io$
Setting the project's status to disabled
displays a maintenance notification to the user. This notification varies for different user roles:
- Admin role sees a popup window (which can be dismissed) and a warning bar at the bottom and can still access all project's views
- Editor, Load data and Viewer roles can access the Project page, but cannot access any of the project's views
So you, as an Admin can still review the changes you've made, but other users are informed about the maintenance and are told to come back later.
Project page for Admin role | Project page for Editor, Load data and Viewer roles |
---|---|
With the live project being disabled, we can truncate it of all data and metadata. Open the project again to close the curent dump, and use the truncateProject
command.
tomas.schmidl@secure.clevermaps.io$ openProject --project k5t8mf2a80tay2ng Project k5t8mf2a80tay2ng (First project) successfully opened tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng$ truncateProject Truncation of project started... Project was successfully truncated. tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng$
Now, we will import the latest dump of the clone project, where we've done all the work. Use importProject
command with both --dump
and --project
parameters.
tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng$ importProject --project c2ovon61uudj9lpm --dump 2019-01-30_17-12-09 Checking if the project k5t8mf2a80tay2ng is empty... OK Importing local dump 2019-01-30_17-12-09 of project c2ovon61uudj9lpm...
Let's skip some of the importProject
command output again...
Import finished! Source project: c2ovon61uudj9lpm (First project (clone)) Source dump: 2019-01-30_17-12-09 Destination project: k5t8mf2a80tay2ng (First project) Destination dump: 2019-01-30_17-23-40 Imported metadata objects: 22 Imported CSV files: 9 To view detailed imported content, use status command. When you're done, use addMetadata command to add the metadata to the project. To push the data to the project, use pushProject command. tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng/dump:2019-01-30_17-23-40$
And as the finish message tells us, use the usual addMetadata
and pushProject
commands to upload the updated dump to the live project.
After that, you as an Admin can open the project in the application. Dismiss the "Maintenance is under way" popup window, enter the Transactions view, and check out if everything works, and the changes you've done in the clone project are there.
Finally, we can switch the project's status back to enabled, and let all our users in.
tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng/dump:2019-01-30_17-23-40$ editProject --project k5t8mf2a80tay2ng --newStatus enabled Project status changed to "enabled" tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng/dump:2019-01-30_17-23-40$