Generate sample data

Generate sample data

In this step we will configure Kinesis Data Generator to generate sample data and push them to Kinesis Firehose

Configure Amazon Cognito for Kinesis Data Generator

  • In this step, we will create a Cloud Formation Stack to configure Cognito. Scripts in CloudFormation will create in Region US East (N. Virginia) us-east-1
  1. Go to AWS Management Console
  • Find CloudFormation
  • Select CloudFormation

Generate Sample Data

  1. In the CloudFormation interface
  • Select Stacks
  • Select Create stack

Generate Sample Data

  1. Download the file cognito-setup.json from First Cloud Journey
  • Use the keyboard shortcut Ctrl + S to Save
  1. In the Create stack interface
  • Select Template is ready
  • Select Upload a template file
  • Select Choose file
  • Select cognito-setup.json
  • Select Next

Generate Sample Data

  1. In the Specify stack details interface
  • Stack name, enter Kinesis-Data-Generator-Cognito-User
  • *User name, enter admin
  • Enter the password you want to use
  • Select Next

Generate Sample Data

  1. Select Next

Generate Sample Data

  1. In the Create stack interface
  • Select I acknowledge that AWS CloudFormation might create IAM resources

Generate Sample Data

  1. Wait about 5 minutes, finish creating stack
  • Select Kinesis-Data-Generator-Cognito-User stack
  • Select Event to see the initialization process

Generate Sample Data

  1. In the Stack interface
  • Select Kinesis-Data-Generator-Cognito-User stack
  • Select Outputs
  • Select KinesisDataGeneratorUrl

Generate Sample Data

  1. In the Amazon Kinesis Data Generator interface, complete the information input (this information you just configured in the Stack creation step)
  • Enter Username
  • Enter Password
  • Select Sign In

Generate Sample Data

  1. In the Amazon Kinesis Data Generator interface, after completing the login
  • Region, select us-east-1
  • Stream/delivery stream, select FCJ-Deliverystream
  • Records per second, enter 2000
  • Select Template 1
  • Enter the following code:
{
  "uuid": "{{random.uuid}}",
  "device_ts": "{{date.utc("YYYY-MM-DD HH:mm:ss.SSS")}}",
  "device_id": {{random.number(50)}},
  "device_temp": {{random.weightedArrayElement(
    {"weights":[0.30, 0.30, 0.20, 0.20],"data":[32, 34, 28, 40]}
  )}},
  "track_id": {{random.number(30)}},
  "activity_type": {{random.weightedArrayElement(
        {
            "weights": [0.1, 0.2, 0.2, 0.3, 0.2],
            "data": ["\"Running\"", "\"Working\"", "\"Walking\"", "\"Traveling\"", "\"Sitting\""]
        }
    )}}
}
  • Then, select Send data

Generate Sample Data

  1. After sending about ~100,000 messages,
  • You can choose Stop Sending Data to Kinesis

Generate Sample Data

  1. Check that the data has gone to S3. Go to AWS Management Console
  • Find S3
  • Select S3

Generate Sample Data

  1. In the S3 interface
  • Select Bucket
  • Select asg-datalake-demo-bucket

Generate Sample Data

  1. In the Bucket interface
  • Select folder data

Generate Sample Data

  1. Then select the raw2022 folder

Generate Sample Data

  1. Thus, the data has gone to S3

Generate Sample Data