Page Comparison

In this tutorial, we'll learn how to use grid visualization with different indicator scales. Currently, there are two ways of using grid in CleverMaps:

connecting your geometryPoint dataset using h3Grid datasets with grid geometries generated on the fly
mapping your geometryPoint dataset using AreaMapper and importing a grid data dimension with vector tiles

This tutorial covers both cases. It's up to you if you want to learn only of them, or both.

Table of Contents

What you'll create

We will add a grid visualization and some more indicator scales.

...

Image Removed

...

Image Removed

...

Image Removed

...

Image Removed

Using H3 grid

CleverMaps natively supports visualization using the H3 grid spatial index developed by Uber. This hierarchical spatial index consists of several resolutions of grid cells which cover the whole Earth, and are related to each other (similar to lower and higher administrative units). The differences between this approach and using AreaMapper and data dimension are:

The geometries are not in vector tile format, but are generated on the fly by the application frontend
There is no need for pre-computed DWH datasets in the project - H3 grids have generated cell names
The grid can be used for any location on Earth (in any resolution)

The usage is also much simpler - to use the H3 grid visualization we need to:

add some h3Grid datasets (each for one resolution)
add these datasets to the ref.h3Geometries array of the geometryPoint dataset we want to visualize

So, let's add these 4 h3Grid datasets:

...

Code Block

title	h3_grid_6.json

{
    "name": "h3_grid_6",
    "type": "dataset",
    "title": "Cell edge 3.2 km (36 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 6,
        "zoom": {
            "min": 2,
            "optimal": 8,
            "max": 18
        }
    }
}

Code Block

title	h3_grid_7.json

{
    "name": "h3_grid_7",
    "type": "dataset",
    "title": "Cell edge 1.2 km (5 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 7,
        "zoom": {
            "min": 2,
            "optimal": 9,
            "max": 18
        }
    }
}

Code Block

title	h3_grid_8.json

{
    "name": "h3_grid_8",
    "type": "dataset",
    "title": "Cell edge 460 m (0.7 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 8,
        "zoom": {
            "min": 2,
            "optimal": 10,
            "max": 18
        }
    }
}

Code Block

title	h3_grid_9.json

{
    "name": "h3_grid_9",
    "type": "dataset",
    "title": "Cell edge 170 m (0.1 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 9,
        "zoom": {
            "min": 2,
            "optimal": 11,
            "max": 18
        }
    }
}

Use addMetadata to add these datasets. Now add the h3Geometries reference to the customers dataset:

Code Block

title	Customers dataset excerpt

   ...
        "ref": {
            "type": "dwh",
            "subtype": "geometryPoint",
            "h3Geometries": [
                "h3_grid_6",
                "h3_grid_7",
                "h3_grid_8",
                "h3_grid_9"
            ],
            "visualizations": [
                {
                    "type": "dotmap"
                },
   ...

And use pushProject to push the changes. You can review the references to the h3Grid datasets in the data model.

Image Removed

Now you can skip right to 7536960, or explore the second way - AreaMapper and data dimension.

Info
Is H3 grid insufficient in performance? The use case described here is a basic one. It's possible to use dataset with materialized H3 grid ID columns which makes the responses up to 2-3x times faster. Contact us at info@clevermaps.io for more information.

Using AreaMapper and data dimension

Running AreaMapper

CleverMaps AreaMapper is one of our data preparation tools. AreaMapper computes geometric intersection between points and polygons (areas, grids) and then adds ID's of polygons to the points table if an intersection exists.

It can be run in Docker or at Keboola. We will use Docker, because not everyone has a Keboola account and it can be run locally.

Info
You can get Docker here. Select Docker Desktop and download the installation file according to your platform.

Then use following command to get the AreaMapper image:

Code Block
docker pull clevermaps/areamapper:latest

We will create an empty directory on our local drive (e.g. /home/user/CleverMaps/AreaMapper) and copy the customers.csv file from the first tutorial into the folder.

Now, we will create a configuration file for AreaMapper. There are two possible ways of running AreaMapper:

using polygons CSV on your local drive
using polygons CSV prepared and hosted by us

The difference is - when using local CSV, you can use any CSV you want. When using the hosted CSV, the file has to be hosted by us first. For more info, contact us at info@clevermaps.io. Current list of hosted dimensions can be found here.

The CSV we will use in this tutorial is already hosted. In case you want to try the "local" way, or just have a look at the CSV, you can download it here.

...

Code Block

title	config-hosted.json

{
    "input_points": {
        "filename": "/workdir/customers.csv",
        "delimiter": ",",
        "geom_fieldname_x": "lng",
        "geom_fieldname_y": "lat",
        "join_fieldname": ""
    },
    "input_areas": {
        "filetype": "candim",
        "filename": "can-dim-grid-uber-9-cz"
    },
    "output": {
        "filename": "/workdir/customers_mapped.csv",
        "delimiter": ",",
        "not_matched_value": ""
    },
    "temp_directory": "/workdir"
}

Code Block

title	config-local.json

{
    "input_points": {
        "filename": "/workdir/customers.csv",
        "delimiter": ",",
        "geom_fieldname_x": "lng",
        "geom_fieldname_y": "lat",
        "join_fieldname": ""
    },
    "input_areas": {
        "filetype": "csv",
        "filename": "/workdir/cz_grid_res9_wkt.csv",
        "geom_fieldname": "wkt",
        "primary_key_fieldname": "hex_id",
        "delimiter": ",",
        "join_fieldname": ""
    },
    "output": {
        "filename": "/workdir/customers_mapped.csv",
        "delimiter": ",",
        "not_matched_value": ""
    },
    "temp_directory": "/workdir"
}

We will run AreaMapper using following command. Please note that you have to modify the paths according to your system.

Code Block

user@host:/home/user/CleverMaps/AreaMapper$ sudo docker run -t -v /home/user/CleverMaps/AreaMapper:/workdir clevermaps/areamapper:latest --config /workdir/config-tutorial-local.json

Importing /workdir/customers.csv to temp database...
Importing /workdir/cz_grid_res9_wkt.csv to temp database...
Creating geometry on points table...
Creating geometry on polygons table...
Creating spatial index on points table...
Creating spatial index on polygon table...
Calculating intersection...
Area mapping was successfully finished.

Copy the output file you specified in the JSON config (customers_mapped.csv) to your project dump /data folder and rename it to customers.csv to overwrite the older customers CSV.

Importing the grid dimension

Let's import the grid data dimension using the importProject command.

Code Block

language	text
theme	Midnight

tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng/dump:2020-06-23_17-48-00$ importProject --project nbutdscdghhg25v4 --server --datasets
Validating DWH model/data integrity of project nbutdscdghhg25v4... OK

Importing project nbutdscdghhg25v4 from server https://secure.clevermaps.io...

Here, we omit a significant amount of the import command output for the sake of readability.

Code Block

language	text
theme	Midnight

Import finished!
Source project: nbutdscdghhg25v4 (can-dim-grid-cz-uber)
Source server: https://secure.clevermaps.io
Destination project: k5t8mf2a80tay2ng (First project)
Destination server: https://secure.clevermaps.io
Imported datasets: 8
Imported metadata objects: 0
Imported CSV files: 4

To view detailed imported content, use status command.
Now you can make changes, or import another project.
When you're done, use addMetadata command to add the metadata to the project.
To push the data to the project, use pushProject command.

Add the imported files using addMetadata and pushProject.

Now we have to edit the customers dataset. AreaMapper added one column to the end of the CSV file - hex_id. We have to add this property to the end of our dataset. It's foreign key will point to the lowest grid dataset - cz_grid_res9_dwh.

Code Block

title	Customers dataset excerpt

  ...
        {
            "name": "lng",
            "title": "Address longitude",
            "column": "lng",
            "type": "longitude",
            "filterable": false
        },
        {
            "foreignKey": "cz_grid_res9_dwh",
            "name": "hex_id",
            "title": "Hex grid ID",
            "column": "hex_id",
            "type": "string",
            "filterable": false
        }
    ],
  ...

Take a look at the data model to see that the customers and cz_grid_res9_dwh are joined.

Image Removed

Grid visualization

Enter the Business overview view, and select the new granularity from the granularity drop down menu.

Image Removed

Now, we have a new visualization available - grid.

Image Removed

Alternatively, select one of the lower levels to get to a bigger detail.

Image Removed

Indicator scales

Let's take a look at more indicator scales. So far we've used only the standard (blue) scale.

To try some of them, set following scales for these indicators:

for online_turnover_indicator, set content.scale to "heatmap"
for offline_turnover_indicator, set content.scale to "magma"
for online_ratio_indicator, set content.scale to "positive"
for offline_ratio_indicator, set content.scale to "negative"

Use pushProject to push the indicators and enter the view. Select the grid visualization, select one of the lower levels (e.g. "Cell edge 460 m") and click "Visualize" on the "Online channel turnover" indicator.

Image Removed

But indicators offer way more scales. Feel free to explore them and use them in your projects!

That's it! The last part of CleverMaps tutorial is a bit more technical and focused on Shell project management

Info

icon	false

Panel

panelIconId	1f389
panelIcon	:tada:
panelIconText	🎉
bgColor	#E3FCEF

The content you are trying to reach has been moved here: https://docs.clevermaps.io/docs/tutorial-8-areamapper-and-grid-visualization

We are proud to announce that we have launched a new documentation.

Please update your saved links and bookmarks to follow a new address docs.clevermaps.io.

Versions Compared

Old Version 18

New Version Current

Key

What you'll create

Using H3 grid

Using AreaMapper and data dimension

Running AreaMapper

Importing the grid dimension

Grid visualization

Indicator scales

The content you are trying to reach has been moved here: https://docs.clevermaps.io/docs/tutorial-8-areamapper-and-grid-visualization