Filter results by

Your first Cloud Connector

Browse source for this tutorial.

In this article we'll show you how to build a Cloud Connector for the Moves cloud. Moves is an Android/iOS app that tracks activity data. After reading this tutorial, you'll understand how to build a Cloud Connector for any cloud that interests you.

You should read Using Cloud Connectors before diving into this tutorial. The Cloud Connector SDK repository includes example Cloud Connector code for other clouds.

Initial configuration

This tutorial assumes you are familiar with ARTIK Cloud and the web tools. If this is your first time using ARTIK Cloud, please see Initial setup in the Web app tutorial.

Create an ARTIK Cloud device type

First, let's configure a new ARTIK Cloud device type as a Cloud Connector.

Create a Moves app

In a new tab, create an application in the Moves Developer Portal to obtain the necessary authentication info.

Take a moment to familiarize yourself with the Moves documentation. Note that Moves uses OAuth 2, like ARTIK Cloud.

  • Log into the Moves Developer Portal.
  • Click on "Manage Your Apps" and then "Create a New App".
  • Give your app a name like "ARTIK Cloud Connector", enter a developer name, and click "Create".
  • Navigate to the Development tab and copy the Client ID and Client Secret. You will need these in the next step.
  • Click "Save Changes".

Configure ARTIK Cloud authentication to Moves cloud

Now return to the ARTIK Cloud Developer Dashboard in the first tab. You should be looking at the "Device Data Source" heading.

  • Navigate to the Cloud Authentication tab under "Device Data Source".
  • Choose "OAuth2" as AUTHENTICATION TYPE.
  • Paste the Client ID and Client Secret you copied from Moves in the previous section.
  • Fill in the following information from the Moves Authentication doc:
    • Authorization URL: https://api.moves-app.com/oauth/v1/authorize
    • Access token URL: POST https://api.moves-app.com/oauth/v1/access_token
    • Credentials Parameters: Expand this section and set external_id to user_id to match the Moves doc.
    • PERMISSION SCOPE: default,activity,location
  • Copy the Redirect URL and Notification URL displayed in Authentication. You will need these in the next step.
  • Save your changes.

Complete Moves setup

Finally, switch back to the Moves Developer Portal. You should be looking at the "Manage Your Apps" page.

  • Navigate to the "Development" tab for your "ARTIK Cloud Connector" app.
  • Fill in the Redirect URL and Notification URL that you copied from ARTIK Cloud in the previous section.

Let's code

Use the Cloud Connector SDK for this section.

Get source code and tools

  • Download and install JDK 8.
  • Clone the Cloud Connector SDK repository from GitHub.
  • Copy and rename template to e.g. movestest.
  • Navigate to the movestest directory.

Prepare Cloud Connector Groovy class

  • Replace the contents of src/main/groovy/com/sample/MyCloudConnector.groovy with MyCloudConnector.groovy at the end of this article.
  • Compile and check: run ../gradlew classes from movestest directory. The command will download the required tools and libraries on demand. Build should be "SUCCESSFUL".

Implementation details

The full Moves Cloud Connector Groovy code MyCloudConnector.groovy is given at the end of this article.

You can use the following libraries to develop the Groovy code:

Below we explain the major methods of MyCloudConnector.groovy. Take note on the following:

subscribe: This method is usually called to subscribe to notifications for this user. In this case, there is no need to implement a subscription because Moves subscribes your Cloud Connector to notifications once you provide a notification URL for your "ARTIK Cloud Connector" app in the Moves Developer Portal (see above).

onNotification: This callback method is triggered when the Moves cloud sends a notification to ARTIK Cloud. This function extracts parameters within the notification. Read the Cloud Connector API documentation to understand the format of the notification parameters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def Or<NotificationResponse, Failure> onNotification(Context ctx, RequestDef req) {
        def json = slurper.parseText(req.content)
        def extId = json.userId.toString()
        def storyLineFiltered = json.storylineUpdates.findAll{ reasonToFetchSummaryData.contains(it.reason) }
        def datesFromStoryLines = storyLineFiltered.collect { e ->
            //We try to recover a valid Date from the storyLine event, and use it to fetch summary data.
            String dtStr = (
                (e.endTime)?e.endTime:
                (e.startTime)?e.startTime:
                (e.lastSegmentStartTime)?e.lastSegmentStartTime:
                null
            )
            DateTime dt = (dtStr != null) ?DateTime.parse(dtStr, receivedDateFormat): DateTime.now()
            requestDateFormat.print(dt)
        }.unique()
        def requestsToDo = datesFromStoryLines.collect{ dateStr ->
            new RequestDef(summaryEndpoint(dateStr)).withQueryParams(queryParams)
        }
        new Good(new NotificationResponse([new ThirdPartyNotification(new ByExternalId(extId), requestsToDo)]))
    }

The above code parses the JSON content received in the request, extracts userId (our external ID that links a Moves user to an ARTIK Cloud device ID), extracts Moves' storyline updates, and extracts date/time to later fetch data from Moves. For each date/time extracted, the code builds a request to fetch data using a base URL (summaryEndpoint) and the date/time.

Next, to fetch data we simply add to the requests built in onNotification the access token we received during authentication. (NOTE: We could also have done this in a signAndPrepare function; refer to Using Cloud Connectors.)

1
2
3
def Or<RequestDef, Failure> fetch(Context ctx, RequestDef req, DeviceInfo info) {
        new Good(req.addHeaders(["Authorization": "Bearer " + info.credentials.token]))
    }

Receiving data

With onFetchResponse we receive some data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def Or<List<Event>, Failure> onFetchResponse(Context ctx, RequestDef req, DeviceInfo info, Response res) {
        switch(res.status) {
            case HTTP_OK:
                def content = res.content.trim()
                if (content == "") {
                    ctx.debug("ignore response valid respond: '${res.content}'")
                    return new Good(Empty.list())
                } else if (res.contentType.startsWith("application/json")) {
                    def json = slurper.parseText(content)
                    def events = json.collectMany { jsData ->
                        def ts = (jsData.date)? getTimestampFromDate(DateTime.parse(jsData.date, requestDateFormat)): ctx.now()
                        extractSummaryNotification(jsData, ts) + extractCaloriesIdle(jsData, ts)
                    }
                    return new Good(events)
                }
                return new Bad(new Failure("unsupported response ${res} ... ${res.contentType} .. ${res.contentType.startsWith("application/json")}"))
            default:
                return new Bad(new Failure("http status : ${res.status} is not OK (${HTTP_OK})"))
        }
    }

In the above code, if the response is a success (HTTP_OK) we parse the JSON response. For each item received, we extract the timestamp and the data, and we create an event with the timestamp and the data (ARTIK Cloud will normalize this data using the Manifest. See About the Manifest for details.)

Test the code locally

Now test the Groovy code, working in the movestest directory unless otherwise specified. (Check Get source code and tools to learn more.)

Unit testing

Replace the contents of src/test/groovy/com/sample/MyCloudConnectorSpec.groovy with MyCloudConnectorSpec.groovy at the end of this article.

Force-run the test by running command ../gradlew cleanTest test. The output of the test should be "BUILD SUCCESSFUL".

Integration testing

You can manually perform integration testing using a local HTTP server. The SDK provides an easy way to run an HTTP (HTTPS) local server. The server runs your Cloud Connector so that you can test authentication and fetching data with the third-party cloud before uploading your code to the ARTIK Cloud Developer Dashboard.

First configure Cloud Connector authentication locally. You should have already configured the authentication to the Moves cloud in the ARTIK Cloud Developer Dashboard. This step is only necessary for testing your Cloud Connector on the local server.

Replace the contents of src/main/groovy/com/sample/cfg.json with the following content. Note that src/main/groovy/com/sample/cfg.json.sample details all availble parameters in the configuration.

1
2
3
4
5
6
7
8
9
10
11
12
13
{
    "authorizationUrl":"https://api.moves-app.com/oauth/v1/authorize",
    "accessTokenUrl":"https://api.moves-app.com/oauth/v1/access_token",
    "clientId":"YOUR_CLIENT_ID_IN_MOVE_DEV_PORTAL",
    "clientSecret":"YOUR_CLIENT_SECRET_IN_MOVE_DEV_PORTAL",
    "authType" : "OAuth2",
    "scope":[ "default", "activity", "location" ],
    "accessTokenUrlMapper":{
        "external_id":"user_id"
    },
    "statusAcceptNotification":200,
    "accessTokenMethod": "post"
}

Replace "YOUR_CLIENT_ID_IN_MOVE_DEV_PORTAL" and "YOUR_CLIENT_SECRET_IN_MOVE_DEV_PORTAL" with the Client ID and Client Secret values you retrieved for your application in the Moves Developer Portal.

Now configure and run Cloud Connector on the local server:

  • Edit src/test/groovy/utils/MyCloudConnectorRun.groovy if you want to change the port of the local server (9080 by default for HTTP and 9083 for HTTPS).
  • NOTE: To receive notifications, your server should be accessible via internet (e.g., use a server accessible from the outside or use an SSH tunnel with port forwarding).
  • The device type ID in the redirect URI is hardcoded to "0000".
  • Temporarily update the configuration in the Moves Developer Portal to use your local server for authentication and notification.
    • Redirect URI: http://localhost:9080/cloudconnectors/0000/auth
    • Notification URI: http://ADDR:9080/cloudconnectors/0000/thirdpartynotifications, replacing ADDR with your server's external address.
  • In the terminal, run the test server ../gradlew runTestServer.
  • Start subscribing to a device by loading http://localhost:9080/cloudconnectors/0000/start_subscription in your Web browser.
  • Follow instructions displayed on the Web page.
  • Generate new data in the Moves app. Then the local server should receive a notification with the new data.
  • In the console, the running test server should print a line "0000: queuing event Event(" for every event. Each event will create an ARTIK Cloud message.

After finishing your integration testing, you should change the configuration on the third-party cloud to use ARTIK Cloud instead of your local test server for authentication and notifications.

If you successfully finish all steps in this section, congratulations! Your Cloud Connector Groovy code passes the test and is ready for submission.

Upload Cloud Connector code to ARTIK Cloud

Log back into the ARTIK Cloud Developer Dashboard and click on your Device Type. Navigate to the Device Info page and find the Device Data Source heading near the bottom.

Switch to the Connector Code tab and copy-and-paste the contents of your src/main/groovy/com/sample/MyCloudConnector.groovy into the text box. You may also upload the file. Finally, save your changes. You are prompted to save the draft or submit it.

Once you submit it, your Cloud Connector code is in approval state. You should hear back from us within 24 hours.

Create device type Manifest

As with all device types, you should create the Manifest for the Cloud Connector device type using either the Simple Manifest or Advanced Manifest workflow. For this particular example, you can upload manifest.json (see below), which we have written and included with the SDK for your convenience.

In the ARTIK Cloud Developer Dashboard, navigate to the Manifest page for your device type. Click on the dropdown icon next to the "+ New Manifest" button:

Upload new Manifest

Here you can upload manifest.json, which has the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
{
  "fields": [
    {
      "children": [
        {
          "name": "activity",
          "type": "CUSTOM",
          "valueClass": "STRING"
        },
        {
          "name": "group",
          "type": "CUSTOM",
          "valueClass": "STRING"
        },
        {
          "name": "duration",
          "type": "CUSTOM",
          "valueClass": "LONG"
        },
        {
          "name": "calories",
          "type": "CUSTOM",
          "valueClass": "LONG"
        },
        {
          "name": "steps",
          "type": "CUSTOM",
          "valueClass": "LONG"
        },
        {
          "name": "distance",
          "type": "CUSTOM",
          "valueClass": "LONG"
        }
      ],
      "name": "summary"
    },
    {
      "name": "caloriesIdle",
      "type": "CUSTOM",
      "valueClass": "Double",
      "unit": "StandardUnits.KILO_CALORIE",
      "isCollection": false,
      "description": "daily idle burn in kcal. Available if user has at least once enabled calories",
      "tags": []
    }
  ],
  "messageFormat": "json",
  "actions": []
}

Because this is a Simple Manifest, it is automatically approved.

About the Manifest

Let's explain the Manifest you uploaded in the above step.

The Moves daily summary API call returns a JSON array of dates and summaries. Each summary JSON object is also a JSON array of activities. Below is an example of the response, which contains two dates and summaries:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[{"date":"20121213",
  "summary":[
     {"activity":"walking","calories":300},
     {"activity":"biking","calories":450}
  ],
  "caloriesIdle":1000,
  "lastUpdate":"20130317T121143Z"
 },
 {"date":"20121214",
  "summary":[
     {"activity": "zumba","duration": 1200, "calories": 500}
  ],
  "caloriesIdle":100,
  "lastUpdate":"20130317T121143Z"}
]

The Cloud Connector Groovy code flattens the top layer of the JSON array into individual JSON objects but drops the date and lastUpdate objects. In addition, each summary JSON array is further flattened into multiple activity JSON objects.

For example, the above response from Moves generates five JSON objects as follows. Each object is the payload of one ARTIK Cloud message. This is our basis for defining manifest.json above. ARTIK Cloud relies on this Manifest to process messages with such payload.

1
2
3
4
5
6
7
8
9
{"summary":{"activity":"walking","calories":300}}

{"summary":{"activity":"biking", "calories":450}}

{"caloriesIdle":1000}

{"summary":{"activity": "zumba","duration": 1200, "calories": 500}}

{"caloriesIdle", 100}

Test Cloud Connector as an ARTIK Cloud user

Once your Cloud Connector is approved, go to My ARTIK Cloud and connect a device using your newly created Cloud Connector device type. Authorize the device to grant ARTIK Cloud access to your Moves data.

Generate new data in your Moves app and synchronize your app to Moves cloud. In My ARTIK Cloud, navigate to the "Data Logs" tab. You should see new data coming into the device you've just connected!

Full Groovy code

MyCloudConnector.groovy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
package com.sample
import org.scalactic.*
import org.joda.time.format.DateTimeFormat
import org.joda.time.*
import groovy.transform.CompileStatic
import groovy.transform.ToString
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
import cloud.artik.cloudconnector.api_v1.*
import static java.net.HttpURLConnection.*
//@CompileStatic
class MyCloudConnector extends CloudConnector {
    static final mdateFormat = DateTimeFormat.forPattern("yyyy-MM-dd").withZoneUTC()
    static final queryParams = ["timeZone": "UTC"]
    static final receivedDateFormat = DateTimeFormat.forPattern("yyyyMMdd'T'HHmmssZ").withZoneUTC().withOffsetParsed()
    static final requestDateFormat = DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC()
    static final reasonToFetchSummaryData = ["DataUpload"]
    JsonSlurper slurper = new JsonSlurper()
    def summaryEndpoint(String date) {
        "https://api.moves-app.com/api/1.1/user/summary/daily/" + date
    }
    @Override
    def Or<NotificationResponse, Failure> onNotification(Context ctx, RequestDef req) {
        def json = slurper.parseText(req.content)
        def extId = json.userId.toString()
        def storyLineFiltered = json.storylineUpdates.findAll{ reasonToFetchSummaryData.contains(it.reason) }
        def datesFromStoryLines = storyLineFiltered.collect { e ->
            //We try to recover a valid Date from the storyLine event, and use it to fetch summary data.
            String dtStr = (
                (e.endTime)?e.endTime:
                (e.startTime)?e.startTime:
                (e.lastSegmentStartTime)?e.lastSegmentStartTime:
                null
            )
            DateTime dt = (dtStr != null) ?DateTime.parse(dtStr, receivedDateFormat): DateTime.now()
            requestDateFormat.print(dt)
        }.unique()
        def requestsToDo = datesFromStoryLines.collect{ dateStr ->
            new RequestDef(summaryEndpoint(dateStr)).withQueryParams(queryParams)
        }
        new Good(new NotificationResponse([new ThirdPartyNotification(new ByExternalId(extId), requestsToDo)]))
    }
    @Override
    def Or<RequestDef, Failure> fetch(Context ctx, RequestDef req, DeviceInfo info) {
        new Good(req.addHeaders(["Authorization": "Bearer " + info.credentials.token]))
    }
    def isSameDay(DateTime d1, DateTime d2) {
        (d1.year== d2.year) && (d1.dayOfYear ==  d2.dayOfYear)
    }
    def isToday(DateTime d) {
        isSameDay(d, DateTime.now())
    }
    def getTimestampOfTheEndOfTheDay(DateTime date) {
        date.minusMillis((date.millisOfDay().get() + 1 )).plusDays(1)
    }
    /**
     * Since we recover the summary data of the day, we want to have a meaningful timestamp from source:
     * If the date is not today : the day is finished. We set the source timestamp to the last second of this past day.
     * If the date is today : the day is not finished, data can continue to evolve for the day, we set the timestamp to now()
     */
    def getTimestampFromDate(DateTime date, DateTimeZone dtz = DateTimeZone.UTC) {
        def now = new DateTime(dtz).toDateTime(dtz)
        def returnedDate = isSameDay(date, now)? now : getTimestampOfTheEndOfTheDay(date)
        returnedDate.getMillis()
    }
    def extractSummaryNotification(jsonNode, long ts) {
        if (jsonNode.summary) {
            jsonNode.summary.collect {js -> new Event(ts, "{\"summary\":" + JsonOutput.toJson(js) + "}")}
        } else {
            []
        }
    }
    def extractCaloriesIdle(jsonNode, long ts) {
        if (jsonNode.caloriesIdle) {
            [new Event(ts,"{\"caloriesIdle\":" + JsonOutput.toJson(jsonNode.caloriesIdle) + "}")]
        } else {
            []
        }
    }
    @Override
    def Or<List<Event>, Failure> onFetchResponse(Context ctx, RequestDef req, DeviceInfo info, Response res) {
        switch(res.status) {
            case HTTP_OK:
                def content = res.content.trim()
                if (content == "") {
                    ctx.debug("ignore response valid respond: '${res.content}'")
                    return new Good(Empty.list())
                } else if (res.contentType.startsWith("application/json")) {
                    def json = slurper.parseText(content)
                    def events = json.collectMany { jsData ->
                        def ts = (jsData.date)? getTimestampFromDate(DateTime.parse(jsData.date, requestDateFormat)): ctx.now()
                        extractSummaryNotification(jsData, ts) + extractCaloriesIdle(jsData, ts)
                    }
                    return new Good(events)
                }
                return new Bad(new Failure("unsupported response ${res} ... ${res.contentType} .. ${res.contentType.startsWith("application/json")}"))
            default:
                return new Bad(new Failure("http status : ${res.status} is not OK (${HTTP_OK})"))
        }
    }
}

MyCloudConnectorSpec.groovy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
package com.sample
import static java.net.HttpURLConnection.*
import utils.FakeContext
import static utils.Tools.*
import spock.lang.*
import org.scalactic.*
import scala.Option
import org.joda.time.format.DateTimeFormat
import org.joda.time.*
import cloud.artik.cloudconnector.api_v1.*
import groovy.json.JsonSlurper
import utils.FakeContext
import static utils.Tools.*
class MyCloudConnectorSpec extends Specification {
    def sut = new MyCloudConnector()
    def parser = new JsonSlurper()
    def ctx = new FakeContext() {
        List<String> scope() {["default", "activity", "location"]}
        long now() { new DateTime(1970, 1, 17, 0, 0, DateTimeZone.UTC).getMillis() }
    }
 
    def apiNotification = '''{ "userId": 23138311640030064,
        "storylineUpdates": [
            {
                "reason": "DataUpload",
                "startTime": "20121312T072747Z",
                "endTime": "20121212T234417-0200",
                "lastSegmentType": "place",
                "lastSegmentStartTime": "20121213T082747Z"
            }
        ]
    }'''
    def apiStoryLineResponse = ''' [
        {
            "date": "20121213",
            "summary": [
                {
                    "activity": "walking",
                    "group": "walking",
                    "duration": 3333,
                    "distance": 3333,
                    "steps": 3333,
                    "calories": 300
                }
            ],
            "segments": [
                {
                    "type": "move",
                    "startTime": "20121212T100051+0200",
                    "endTime": "20121212T100715+0200",
                    "activities": [
                        {
                            "activity": "walking",
                            "group": "walking",
                            "manual": false,
                            "startTime": "20121212T100051+0200",
                            "endTime": "20121212T100715+0200",
                            "duration": 384,
                            "distance": 421,
                            "steps": 488,
                            "calories": 99,
                            "trackPoints": [
                                {
                                    "lat": 55.55555,
                                    "lon": 33.33333,
                                    "time": "20121212T100051+0200"
                                },
                                {
                                    "lat": 55.55555,
                                    "lon": 33.33333,
                                    "time": "20121212T100715+0200"
                                }
                            ]
                        }
                    ],
                    "lastUpdate": "20130317T121143Z"
                }
            ],
            "caloriesIdle": 1000,
            "lastUpdate": "20130317T121143Z"
        }
    ]'''
 
    def extId = "23138311640030064"
    def apiEndpoint = "https://api.moves-app.com/api/1.1"
    def device = new DeviceInfo("deviceId", Option.apply(extId), new Credentials(AuthType.OAuth2, "", "1j0v33o6c5b34cVPqIiB_M2LYb_iM5S9Vcy7Rx7jA2630pK7HIjEXvJoiE8V5rRF", Empty.option(), Option.apply("bearer"), ctx.scope(), Empty.option()), ctx.cloudId(), Empty.option())
    // def "add accessToken into requests about device, with Credentials"() {
    //     //nothing to do
    // }
    def "build requests on notification"(){
        when:
        def req = new RequestDef("").withMethod(HttpMethod.Post).withContent(apiNotification, "application/json")
        println("req.get "+req)
        def res = sut.onNotification(ctx, req)
        then:
        res.isGood()
        res.get() == new NotificationResponse([
            new ThirdPartyNotification(new ByExternalId(device.extId.get()), [
                new RequestDef(apiEndpoint + "/user/summary/daily/20121213").withQueryParams(["timeZone": "UTC"]),
            ]),
        ])
    }
    def "fetch summary 20121213"() {
        when:
        def req = new RequestDef(apiEndpoint + "/user/summary/daily/20121213").withQueryParams(["timeZone": "UTC"])
        def resp = new Response(HttpURLConnection.HTTP_OK, "application/json", apiStoryLineResponse)
        def res = sut.onFetchResponse(ctx, req, device, resp)
        then:
        res.isGood()
        //20121213 at 23h59:59 = 1355356799 seconds
        def timestamp_20121213=1355443199999L
        def events = [
            new Event(timestamp_20121213, """{"summary":{"activity":"walking","group":"walking","duration":3333,"distance":3333,"steps":3333,"calories":300}}"""),
            new Event(timestamp_20121213, """{"caloriesIdle":1000}""")
        ]
        cmpEvents(res.get(), events)
    }
}