Generate sample data
Generate sample data
In this step we will configure Kinesis Data Generator to generate sample data and push them to Kinesis Firehose
Configure Amazon Cognito for Kinesis Data Generator
- In this step, we will create a Cloud Formation Stack to configure Cognito. Scripts in CloudFormation will create in Region US East (N. Virginia) us-east-1
- Go to AWS Management Console
- Find CloudFormation
- Select CloudFormation

- In the CloudFormation interface
- Select Stacks
- Select Create stack

- Download the file cognito-setup.json from First Cloud Journey
- Use the keyboard shortcut Ctrl + S to Save
- In the Create stack interface
- Select Template is ready
- Select Upload a template file
- Select Choose file
- Select cognito-setup.json
- Select Next

- In the Specify stack details interface
- Stack name, enter
Kinesis-Data-Generator-Cognito-User
- *User name, enter
admin
- Enter the password you want to use
- Select Next

- Select Next

- In the Create stack interface
- Select I acknowledge that AWS CloudFormation might create IAM resources

- Wait about 5 minutes, finish creating stack
- Select Kinesis-Data-Generator-Cognito-User stack
- Select Event to see the initialization process

- In the Stack interface
- Select Kinesis-Data-Generator-Cognito-User stack
- Select Outputs
- Select KinesisDataGeneratorUrl

- In the Amazon Kinesis Data Generator interface, complete the information input (this information you just configured in the Stack creation step)
- Enter Username
- Enter Password
- Select Sign In

- In the Amazon Kinesis Data Generator interface, after completing the login
- Region, select us-east-1
- Stream/delivery stream, select FCJ-Deliverystream
- Records per second, enter 2000
- Select Template 1
- Enter the following code:
{
"uuid": "{{random.uuid}}",
"device_ts": "{{date.utc("YYYY-MM-DD HH:mm:ss.SSS")}}",
"device_id": {{random.number(50)}},
"device_temp": {{random.weightedArrayElement(
{"weights":[0.30, 0.30, 0.20, 0.20],"data":[32, 34, 28, 40]}
)}},
"track_id": {{random.number(30)}},
"activity_type": {{random.weightedArrayElement(
{
"weights": [0.1, 0.2, 0.2, 0.3, 0.2],
"data": ["\"Running\"", "\"Working\"", "\"Walking\"", "\"Traveling\"", "\"Sitting\""]
}
)}}
}

- After sending about ~100,000 messages,
- You can choose Stop Sending Data to Kinesis

- Check that the data has gone to S3. Go to AWS Management Console

- In the S3 interface
- Select Bucket
- Select asg-datalake-demo-bucket

- In the Bucket interface

- Then select the raw2022 folder

- Thus, the data has gone to S3
