How to use the Kinesis Connector

Integrate and extend your AirVantage platform

Here we give you an overview of what you can do with the AirVantage Cloud Connector via a simple use case: Connecting your system to AWS Kinesis and getting raw data from your Kinesis stream.

We’ll explain all the technical aspects needed for a robust, secure & reliable interface:

principles & best practices

configuration & authentication

how to get messages

1 / Principles

2 / Configuration

3 / Get messages

Principles

This section explains when and why to use the Cloud connector, what it is and the available options.

Introduction

The Kinesis Cloud Connector provides:

message notifications from AirVantage
Robust messaging for applications
High volume messaging
Runs on all major operating systems
Open source libraries for several common languages (java, c#, python, PHP, …)

Parameters

When you connect AirVantage to Kinesis Streams, you have to define the type of notifications you want: operations, data, alert or usages.

Be aware that Kinesis acknowledges a set of messages and not all individual messages.

Configuration

This section explains how to configure your client to be robust, reliable and secure. Please read carefully this section in order to supply correct information to enable the end point on the AirVantage side.

What can I use?

AWS SDK for Java or KCL to start developing your application in your favorite language.

AWS SDK is usefull to get started with Amazon Kinesis Streams, but you have to use KCL to build a complete application that can connect to Kinesis and process received data.

Prerequisites

Set up an Amazon web services account here .

Introduction

The role describes the rights (called policies) allowed for an AWS account on a specific object (here a Kinesis sream).

In the schema on the right, we created an airvantageKinesisProducer role applied to AirVantage AWS account for a given company defined by the company uid in the secretKey.

Allowed policies are the ones allowing to push messages in the stream and the last parameter defines the Kinesis Stream to use.

Configuration Process

Step 1. Create a Kinesis stream in the customer’s AWS account, into which AirVantage will publish.

Be carefull ! There is a charge for using Kinesis streams resources. So before creating your stream, calculate the number of shards you need according to the quantity of data you handle and estimate your corresponding bill.

Step 2. Create a Role in the customer’s AWS account:

The Role Type should be Cross-Account Access > 3rd party AWS account.
The Account ID should be 406826153012 (Sierra’s own AWS account ID).
The External ID should be the UID of the customer’s company in AirVantage.
In the role policy, click on Next and then Finish
Edit the role in order to add an Inline policies: The role’s Policy should only allow Sierra to write into the Kinesis stream (kinesis:PutRecord, kinesis:PutRecords).

In case of you are not aware how to do this last step using the AWS Console, please have a look to How to configure my stream?

How to configure my stream?

Here is a detailed howto using AWS CLI :

Edit the following Role-Trust-Policy.json replace by the AirVantage’s company id of the company which we want to receive data from. In case of multiple AirVantag’s companies, multiple roles and CloudConnector configuration must be created.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Condition": { "StringEquals": { "sts:ExternalId": "<AIRVANTAGE COMPANY ID>" } },
      "Principal": { "AWS": "arn:aws:iam::406826153012:root" }
    }
  ]
}

and execute:

# aws iam create-role --role-name sierra --assume-role-policy-document file:///.../Role-Trust-Policy.json

AWS response should look like :

{
  "Role": {
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Effect": "Allow",
        "Condition": {
          "StringEquals": {
            "sts:ExternalId": "<AIRVANTAGE COMPANY ID>"
          }
        },
        "Principal": {
          "AWS": "arn:aws:iam::406826153012:root"
        }
      }
      ]
    },
    "RoleId": "<ROLE ID>",
    "CreateDate": "2016-09-12T12:16:16.727Z",
    "RoleName": "sierra",
    "Path": "/",
    "Arn": "arn:aws:iam::<AWS ACCOUNT ID>:role/sierra"
  }
}

Then edit the following Role-Policy.json and replace with your AWS account id and with the Kinesis stream delivery name.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [
        "arn:aws:kinesis:eu-west-1:<AWS ACCOUNT ID>:stream/<STREAM NAME>"
      ],
      "Effect": "Allow"
    }
  ]
}

and execute:

# aws iam put-role-policy --role-name sierra --policy-name kinesis_push --policy-document file:///.../Role-Policy.json

No response is expected from this command.

Note: during the creation of a Kinesis delivery stream with the AWS console an IAM role named kinesis_delivery_role is created. This role should not be added a Trusted relationship with AirVantage.

Step 3. Supply the following information using the CRM and select Request Support:

Customer’s AWS Account ID (a 12-digit number).
Name of the Role they created for us.
Customer’s AWS Region ID (e.g. us-east-1).
Name of the Kinesis stream we should write into.
Kind of messages you want to receive:
- Alert event: any alert rule triggered will be send
- Operation: any operation state will be sent (creation, progress, success or failed)
- New message sent by the device: any device incoming communication
- Usages: any SIM usage
- New message sent by the device: any device incoming communication
  - Optionnaly you can supply the data path to whitelist the data (only the values for these data will be sent in the data stream). To get the data path for a data, you can go to the timeline to view the data path in the tooltip like in the screenshot below:

Test your setup

You can check that the data received from AirVantage shows up in the stream monitoring provided by Amazon. The graph that describes the number of Put Record Requests lets you ensure that your Kinesis stream is connected to AirVantage. If no data is shown, you need to check your configuration.

Get the messages

This section explains how to consume the messages from AWS Kinesis in order to support high volume messaging.

Using Kinesis Client Library

KinesisStreamApp : This Kinesis stream java application uses KCL to connect to Kinesis Streams and pulls messages.

Installation

Configure the AWS CLI: aws configure is the command to type at the command line. It is an interactive command, you need to provide the following four pieces of information: AWS Access Key ID, AWS Secret Access Key, Default region, and Default output format.
```
     aws configure
```
Clone the application using git from the repository.

You can optionally use eclipse to edit the files as an Eclipse project is supplied.

The source project is organized with three main files:

A configuration file to edit (see next section)
A Record Processor that processes records
The main application. This connects to the stream by creating a worker that pulls records, makes checkpoints and displays data on the console.

Configuration

In the configuration file config.properties, you have to specify:

Unique Application name (a DynamoDB table is created for each application to save its state, such as checkpoints).
Number of records to receive to make a checkpoint
Maximum number of records returned by one call
Stream name
Region (for instance us-west-2)

How to run

The configuration file config.properties has to be adapted.

Enjoy !

Test

Check that the application displays the messages in the console. In the CloudWatch Stream Monitoring, the Get Requests graph describes the evolution of Get Requests over time. And this shows that your application is successfully connected to Kinesis and is retrieving messages.

Samples

KinesisStreamApp
Other languages: You can find clients in various languages supplied by kinesis

Matching results

1 / Principles

2 / Configuration

3 / Get messages

Principles

Introduction

Parameters

Configuration

What can I use?

Prerequisites

Introduction

Configuration Process

Test your setup

Get the messages

Using Kinesis Client Library

Installation

Configuration

How to run

Test

Samples