mirror of
https://gitlab.ewi.tudelft.nl/ee2l1/2025-2026/A.K.03.git
synced 2025-12-12 15:30:57 +01:00
730 lines
37 KiB
Plaintext
730 lines
37 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"\n",
|
||
"[Table of Contents](0_Table_of_Contents.ipynb)\n",
|
||
"\n",
|
||
"# Chapter 4: Module 2 - Reading KITT Sensor Data\n",
|
||
"\n",
|
||
"**Contents:**\n",
|
||
"* [Distance Sensor](#distance-sensors)\n",
|
||
"* [The Microphones](#the-microphones)\n",
|
||
"* [FAQ](#faq)\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Import necessary libraries\n",
|
||
"import time\n",
|
||
"import numpy as np\n",
|
||
"import pandas as pd\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"import csv\n",
|
||
"\n",
|
||
"# Uncomment one of the following lines depending on your setup\n",
|
||
"\n",
|
||
"# If you are using the real car, uncomment the next lines and comment the simulator lines\n",
|
||
"# from serial import Serial\n",
|
||
"# import sounddevice\n",
|
||
"\n",
|
||
"# If you are using the simulator, uncomment the next lines and comment the real car lines\n",
|
||
"from KITT_Simulator.serial_simulator import Serial\n",
|
||
"from KITT_Simulator.sounddevice_simulator import sounddevice\n",
|
||
"\n",
|
||
"# Note: After changing the import statement, you need to restart the kernel for changes to take effect."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"KITT relies on its sensors to drive autonomously. It is equipped with:\n",
|
||
"1. Two front-mounted distance sensors.\n",
|
||
"2. Five microphones positioned around the field to record audio signals from KITT's beacon and relay them to the soundcard, after which they can be read by your PC.\n",
|
||
"\n",
|
||
"This task focuses on reading data from the distance sensors to avoid obstacles and processing the microphone data from the field.\n",
|
||
"\n",
|
||
"**Preparation**\n",
|
||
"- Ensure KITT is operational and properly set up.\n",
|
||
"- Reserve a time slot for testing on a field equipped with microphones and an audio card.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Distance Sensors\n",
|
||
"\n",
|
||
"KITT’s front distance sensors use ultrasonic technology. Two SRF02 modules, mounted on the left and right sides, measure the distance to obstacles. These \"parking sensors\" work by emitting a 40 kHz pulse and measuring the time it takes for the echo to return. This time is converted into a distance measurement.\n",
|
||
"\n",
|
||
"- Each sensor requires a minimum of 66 ms between readings, as specified in the SRF02 datasheet (available on Brightspace or at `Files/Datasheets/srf02.pdf`).\n",
|
||
"- The system is configured with a 70 ms cycle time; the left and right sensors take turns recording measurements.\n",
|
||
"- These measurements are stored in a buffer on KITT's microcontroller, with each new reading overwriting the previous one. When you request the KITT status, you will obtain a copy of the current buffer values.\n",
|
||
"\n",
|
||
"<img src=\"pictures/srf02-ultrasonic-sensor.jpg\" alt=\"Ultrasonic Sensor\" width=\"400\" height=\"240\">\n",
|
||
"\n",
|
||
"### Step 0: Characteristics of the Distance Sensors\n",
|
||
"\n",
|
||
"Using the readings on the car display, report on the following questions:\n",
|
||
"\n",
|
||
"1. What is the accuracy of the distance sensors? Does this change with distance?\n",
|
||
"2. What are the minimum and maximum distances the sensors can measure?\n",
|
||
"3. What is the field of view of the distance sensors (beam angle)?\n",
|
||
"\n",
|
||
"To measure this field of view, move an obstacle from left to right over a line, at about 1 m distance from the sensors, and observe when the sensors start to 'see' the object.\n",
|
||
"\n",
|
||
"The field of view is important when making recordings: you should realize that the distance sensors may detect chairs, bags, etc., and then make false readings. This happens even if these objects are not straight in front of the sensors. (The field of view does depend on distance.)\n",
|
||
"\n",
|
||
"**Note:** Do not copy the questions into your report, but naturally include the information in your report as part of your discussion.\n",
|
||
"\n",
|
||
"### Step 1: Status Command\n",
|
||
"\n",
|
||
"To ensure you can experiment at home, we have added the status command to the simulator. The simulator will accurately simulate the sensor distances, but not its behavior. Make sure to test on the real car frequently.\n",
|
||
"\n",
|
||
"As you have learned in the previous module, you can ask KITT to capture a status command by writing `\"S\\n\"` to the serial port. Then you have to read the message using `read_until`; this will generate a binary message that you need to decode. KITT always ends its message with the end-of-transmission character (0x04). The response contains three sections:\n",
|
||
"1. Audio beacon status and settings\n",
|
||
"2. PWM values for the motors\n",
|
||
"3. Sensor readings"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n",
|
||
"\n",
|
||
"# TODO: Establish a serial connection, ask for a status report, read it out, and print it\n",
|
||
"\n",
|
||
"# TODO: Close the serial connection"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"If you only need distance measurement information, you can request it separately:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"serial.write(b'Sd\\n')\n",
|
||
"```\n",
|
||
"\n",
|
||
"This returns only the left and right distance sensor values, filtering out the rest of the status report.\n",
|
||
"\n",
|
||
"### Step 2: Extracting and Isolating Distance Data\n",
|
||
"\n",
|
||
"Assuming you have received the full status information from KITT, you can extract and isolate the distance sensor readings (left and right) from the status report.\n",
|
||
"\n",
|
||
"After sending the status command (`b'S\\n'`), the response will contain a variety of information, including the distance measurements. Now write a Python function to extract the distance data from the status report.\n",
|
||
"\n",
|
||
"1. **Extract the distance measurements**:\n",
|
||
"\n",
|
||
"The distance values are typically embedded in the `Sensors` section of the status response. You can process the `status` output to isolate just the left (`L`) and right (`R`) distance sensor values. Write a function to extract these values.\n",
|
||
"\n",
|
||
"*Hints:*\n",
|
||
"\n",
|
||
"- Use `decode('utf-8')` to convert bytes to a string.\n",
|
||
"- Use `splitlines()` to separate the status message into individual lines.\n",
|
||
"- Look for the line that contains `\"Dist.\"` to find the distance measurements.\n",
|
||
"- Use `split()` to break the line into individual words.\n",
|
||
"- Be cautious of the positions of the distance values in the list; adjust indices as necessary.\n",
|
||
"\n",
|
||
"**Note:** If you use the `Sd` status command, you retrieve less info and can write a faster function! The parsing of the status string is also easier."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n",
|
||
"\n",
|
||
"# TODO: Decode the status response to a string\n",
|
||
"\n",
|
||
"# TODO: Split the status string into lines\n",
|
||
"lines = \n",
|
||
"\n",
|
||
"# Initialize variables to hold distance values\n",
|
||
"dist_L = None\n",
|
||
"dist_R = None\n",
|
||
"\n",
|
||
"# Iterate over each line to find distance data\n",
|
||
"for line in lines:\n",
|
||
" if \"Dist.\" in line:\n",
|
||
" # TODO: Split the line into words\n",
|
||
" words =\n",
|
||
" # Extract distance values based on their positions\n",
|
||
"\n",
|
||
" # Assign dist_L and dist_R accordingly\n",
|
||
" dist_L = \n",
|
||
" dist_R = \n",
|
||
" break # Exit the loop after finding the distances\n",
|
||
"\n",
|
||
"# Print the extracted distance values\n",
|
||
"print(f\"Left Distance: {dist_L}\")\n",
|
||
"print(f\"Right Distance: {dist_R}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"2. **Determine how fast you can read out (and process) your distance data** by writing a script that requests the status 100 times. You can calculate the average delay for this operation (and its standard deviation); you could also present the results in a histogram. To measure delays, you will need to keep track of time:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"start_time = time.time() # Initialize\n",
|
||
"current_time = time.time() - start_time # Find current time since initialization\n",
|
||
"```\n",
|
||
"\n",
|
||
"If you can read out the sensors faster than 70 ms (or is it 140 ms?), reason if you will obtain duplicate values from the buffer.\n",
|
||
"\n",
|
||
"**Student Task:**\n",
|
||
"\n",
|
||
"- Write a script that sends the status command 100 times, recording the time taken for each read.\n",
|
||
"- Store the time intervals in a list.\n",
|
||
"- After collecting the data, calculate the average delay and standard deviation.\n",
|
||
"- Plot a histogram of the delays."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 3: Using Distance Values to Model the Car\n",
|
||
"\n",
|
||
"Ultrasonic sensors are not just used for detecting obstacles; they play a crucial role in modeling the car's behavior during autonomous driving. To control the car effectively, we need to understand how it responds to drive and steering commands, similar to how a human driver knows how much acceleration or steering input affects the car's movement.\n",
|
||
"\n",
|
||
"However, while KITT doesn’t have an accelerometer to measure acceleration directly, we can use the ultrasonic sensors to estimate how the car moves over time. By measuring the distance to a cardboard-box wall, we can derive its speed and acceleration.\n",
|
||
"\n",
|
||
"#### Understanding Speed and Acceleration\n",
|
||
"\n",
|
||
"- **Velocity** is the rate of change of position over time:\n",
|
||
"\n",
|
||
" $$\n",
|
||
" v(t) = \\frac{{\\mathrm d} x}{{\\mathrm d} t}(t) \\,.\n",
|
||
" $$\n",
|
||
"\n",
|
||
"- **Acceleration** is the change in speed over time:\n",
|
||
"\n",
|
||
" $$\n",
|
||
" a(t) = \\frac{{\\mathrm d} v}{{\\mathrm d} t}(t) \\,.\n",
|
||
" $$\n",
|
||
"\n",
|
||
"Note that $x(t)$, $v(t)$ and $a(t)$ are time varying. To implement the differentials, in practice we will subtract two subsequent samples $x(t_1)$ and $x(t_2)$. We will then have an estimate of $v(t)$ for $t = (t_1+t_2)/2$:\n",
|
||
"\n",
|
||
"$$\n",
|
||
"v\\left(\\frac{t_1+t_2}{2}\\right) \\approx \\frac{x(t_2) - x(t_1)}{t_2-t_1} \\,.\n",
|
||
"$$\n",
|
||
"\n",
|
||
"In theory, this approximation gets better for $t_2$ close to $t_1$, but at the same time the division of two small numbers will make the result very sensitive to noise, so in practice there is a trade-off.\n",
|
||
"\n",
|
||
"#### Plotting KITT's Motion Towards a Wall\n",
|
||
"\n",
|
||
"To understand how KITT moves, make recordings of the distance sensor values as KITT drives towards a wall. Do this for multiple motor commands, and store them in a `.csv` file. (You can use the `Files/Recordings` folder to organize your data). You can then later import the data into Python. Next to the sensor values, you should also store the time stamp of each sample.\n",
|
||
"\n",
|
||
"You will see KITT speed up, and then reach a constant speed. To do this experiment, please let KITT drive towards the supplied cardboard wall. **Turn off KITT's motors once the distance is less than 40 cm to ensure KITT does not crash into the wall.** Note that you may have to discard the first few readings as they may be inaccurate.\n",
|
||
"\n",
|
||
"*Hints:*\n",
|
||
"\n",
|
||
"- Choose an appropriate motor speed value for `motor_speed_value`.\n",
|
||
"- Ensure that you stop the car if it gets too close to the wall to prevent collisions.\n",
|
||
"- Use `time.time()` to keep track of elapsed time.\n",
|
||
"- Store the data in a list with the format `[current_time, dist_L, dist_R]`.\n",
|
||
"- Write the data to a CSV file for later analysis.\n",
|
||
"- Also document the motor speed setting (e.g. use this as part of your file name)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n",
|
||
"\n",
|
||
"# TODO: Open the serial connection to KITT, set the motor speed\n",
|
||
"\n",
|
||
"# Initialize a list to store recorded data\n",
|
||
"data = []\n",
|
||
"\n",
|
||
"# Record data for a specified duration (e.g., 10 seconds)\n",
|
||
"recording_duration = 10 # in seconds\n",
|
||
"start_time = time.time()\n",
|
||
"\n",
|
||
"while time.time() - start_time < recording_duration:\n",
|
||
" # Send the status command to get the distance readings\n",
|
||
" serial_port.write(b'S\\n')\n",
|
||
" \n",
|
||
" # Read the status response\n",
|
||
" status = serial_port.read_until(b'\\x04').decode('utf-8')\n",
|
||
" \n",
|
||
" # TODO: Extract the distance values from the status response\n",
|
||
"\n",
|
||
" dist_L = None\n",
|
||
" dist_R = None\n",
|
||
" \n",
|
||
" # TODO: Record current time and distances\n",
|
||
" data.append([current_time, dist_L, dist_R])\n",
|
||
" \n",
|
||
" # Check if KITT is too close to the wall and stop if necessary\n",
|
||
" if dist_L < 40 or dist_R < 40:\n",
|
||
" serial_port.write(b'M150\\n') # Stop the car\n",
|
||
" print(\"Stopping KITT to avoid collision.\")\n",
|
||
" break # Exit the loop\n",
|
||
" # Note: you can also add a small loop here and still read the stopping data\n",
|
||
" \n",
|
||
" time.sleep(0.1) # Wait before the next reading\n",
|
||
"\n",
|
||
"# Close the serial connection\n",
|
||
"serial_port.close()\n",
|
||
"\n",
|
||
"# TODO: Write the recorded data to a CSV file\n",
|
||
"# Recommeded file output: Files/Recordings/kitt_distance_data_{speed}.csv"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Processing the Recorded Data\n",
|
||
"\n",
|
||
"Read your `.csv` data into Python (see template script below), and plot the distance values over time to visualize KITT's motion. Use a single plot with separate colors for the L and R sensors. You should notice a 'staircase' shape! Explain this in your report.\n",
|
||
"\n",
|
||
"Next, to derive velocity, first merge the L and R sensor data into a single position estimate: remove the duplicate values (keep only the first value of a duplicate reading), then merge the remaining values into a single (time, position) array. Plot the result in your distance plot to see if you did this correctly.\n",
|
||
"\n",
|
||
"After that, estimate the velocity of KITT as function of time. Obviously, you will use $ v(t) = \\Delta x / \\Delta t \\ $, but what time $t$ do you associate with each of these estimates?\n",
|
||
"\n",
|
||
"Make a plot of the resulting velocity estimates over time.\n",
|
||
"\n",
|
||
"*Hints:*\n",
|
||
"\n",
|
||
"- When merging the L and R distance measurements, don't simply average them. Read the above paragraph again.\n",
|
||
"- Be aware that the sensors alternate readings every 70 ms, leading to a 'staircase' effect.\n",
|
||
"- To calculate velocity, use the differences in distance and time (`diff()` function).\n",
|
||
"- Since KITT is moving towards the wall, the distance decreases; yet, the estimated velocity should be positive when moving forward.\n",
|
||
"- For the time associated with each velocity estimate, use the midpoint between consecutive time stamps.\n",
|
||
"- Remove any NaN values resulting from the `diff()` operation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n",
|
||
"\n",
|
||
"motor_speed_value = 160 # Use the same motor speed as during recording\n",
|
||
"\n",
|
||
"# Load the recorded data from the CSV file\n",
|
||
"csv_filename = f'Files/Recordings/kitt_distance_data_{motor_speed_value}.csv'\n",
|
||
"data = pd.read_csv(csv_filename)\n",
|
||
"\n",
|
||
"# Consider discarding the first few readings (inaccurate readings)\n",
|
||
"\n",
|
||
"# TODO: Remove duplicate time stamps and merge L and R data\n",
|
||
"# Create a new DataFrame to hold your processed data\n",
|
||
"merged_data = []\n",
|
||
"\n",
|
||
"# Iterate over the data\n",
|
||
"for index, row in data.iterrows():\n",
|
||
" # Extract time and distances\n",
|
||
" time_stamp = row['Time']\n",
|
||
" dist_L = row['Distance_L']\n",
|
||
" dist_R = row['Distance_R']\n",
|
||
" \n",
|
||
" # TODO: Decide which distance to use or how to merge them\n",
|
||
" distance = \n",
|
||
"\n",
|
||
" merged_data.append([time_stamp, distance])\n",
|
||
"\n",
|
||
"# Convert merged data to DataFrame\n",
|
||
"merged_df = pd.DataFrame(merged_data, columns=['Time', 'Distance'])\n",
|
||
"\n",
|
||
"# TODO: calculate velocity (change in distance over change in time)\n",
|
||
"merged_df['Velocity'] = \n",
|
||
"\n",
|
||
"# Note: Use a negative sign because distance to the wall decreases as KITT moves forward\n",
|
||
"\n",
|
||
"# Calculate the time corresponding to each velocity estimate\n",
|
||
"# It's common to use the midpoint of the time intervals\n",
|
||
"merged_df['Velocity_Time'] = merged_df['Time'] - merged_df['Time'].diff() / 2\n",
|
||
"\n",
|
||
"# Plotting Distance\n",
|
||
"plt.figure()\n",
|
||
"plt.plot(merged_df['Time'], merged_df['Distance'], label='Distance to Wall')\n",
|
||
"plt.xlabel('Time (s)')\n",
|
||
"plt.ylabel('Distance (cm)')\n",
|
||
"plt.title('Distance to Wall Over Time')\n",
|
||
"plt.grid(True)\n",
|
||
"plt.legend()\n",
|
||
"plt.show()\n",
|
||
"\n",
|
||
"# Plotting Velocity\n",
|
||
"plt.figure()\n",
|
||
"plt.plot(merged_df['Velocity_Time'], merged_df['Velocity'], label='Velocity (cm/s)')\n",
|
||
"plt.xlabel('Time (s)')\n",
|
||
"plt.ylabel('Velocity (cm/s)')\n",
|
||
"plt.title('Velocity of KITT Over Time')\n",
|
||
"plt.grid(True)\n",
|
||
"plt.legend()\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Tips to Consider:**\n",
|
||
"\n",
|
||
"**Continuous Measurement** involves data that can be taken at any point in time, with no gaps. For example, a car’s speedometer provides a continuous record of the car’s speed.\n",
|
||
"\n",
|
||
"**Discrete Measurement**, on the other hand, collects data at specific intervals. For instance, KITT’s ultrasonic sensors take distance readings every 70 ms. In between these measurements, we don’t know the exact position of the car. Discrete data can still be useful, but it may miss details about rapid changes in speed or acceleration that occur between measurements. In order to interpret it correctly, you may need to filter or interpolate the data.\n",
|
||
"\n",
|
||
"The following shows the difference between continuous and discrete data:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Define time for continuous measurement (smooth, no gaps)\n",
|
||
"time_continuous = np.linspace(0, 10, 1000) # Time from 0 to 10 seconds, 1000 data points\n",
|
||
"# Define time for discrete measurement (specific intervals)\n",
|
||
"time_discrete = np.linspace(0, 10, 20) # Time from 0 to 10 seconds, 20 data points\n",
|
||
"# Simulate continuous speed (sinusoidal speed pattern for illustration)\n",
|
||
"speed_continuous = 10 * np.sin(0.5 * np.pi * time_continuous) # Continuous speed\n",
|
||
"# Simulate discrete speed (sampled at specific intervals)\n",
|
||
"speed_discrete = 10 * np.sin(0.5 * np.pi * time_discrete) # Discrete speed\n",
|
||
"# Plotting both continuous and discrete measurements\n",
|
||
"plt.figure(figsize=(7, 4))\n",
|
||
"plt.plot(time_continuous, speed_continuous, label=\"Continuous Measurement\", color=\"blue\")\n",
|
||
"plt.scatter(time_discrete, speed_discrete, label=\"Discrete Measurement\", color=\"red\", zorder=5)\n",
|
||
"plt.xlabel('Time (s)')\n",
|
||
"plt.ylabel('Speed (cm/s)')\n",
|
||
"plt.title('Continuous vs Discrete Measurement of Speed')\n",
|
||
"plt.legend()\n",
|
||
"plt.grid(True)\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 5: Implementing the Distance Sensor Reading in Your KITT Class\n",
|
||
"\n",
|
||
"In the previous module, you have created a class for KITT. Add a method to read the distance sensors to your `KITT` class in your 'Student Code' files. You can use the code you have written in the previous steps to do this. Make sure to test your code. It is advisable to store all the old distance data in a list inside the `KITT` class. This will be convenient during the final challenge, where the route planning might need old measurements to determine the position of objects.\n",
|
||
"\n",
|
||
"**Student Task:**\n",
|
||
"\n",
|
||
"- Add a method `read_distance_sensors()` to your `KITT` class.\n",
|
||
"- The method should send the status command to KITT and extract the distance measurements.\n",
|
||
"- Store the readings along with timestamps in an internal list or data structure.\n",
|
||
"\n",
|
||
"### Step 6: Mid-term Assessment 2.1 and Report\n",
|
||
"\n",
|
||
"After you finish this assignment, and ultimately in week 4, showcase the functionality of your script to your assigned TA. After you pass this assessment, you are ready to document your results in your midterm report. For this Module, you would include a chapter that covers the above tasks (using independently-readable text, i.e., don’t refer to “Step 1”).\n",
|
||
"\n",
|
||
"Include plots; for each plot, it should be clear how the plot was made (i.e., the corresponding experimental setup), and you have to describe what is seen in the plot before you discuss results and derive conclusions. Review the guidelines in Chapter 7 for more information. Include the corresponding code in an appendix.\n",
|
||
"\n",
|
||
"Remember to document your code, using comments to define input/output variables of functions and to explain the logic and any modifications made. Your completed script will be crucial for the upcoming challenges, contributing to the overall autonomous driving system."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## The Microphones\n",
|
||
"\n",
|
||
"The field is equipped with four microphones at its corners, and a fifth microphone positioned at a higher level between two of the edge microphones. These microphones, together with the beacon mounted on KITT, will be used to locate KITT within the field (more details in Chapter 5).\n",
|
||
"\n",
|
||
"<img src=\"pictures/axisdef.png\" alt=\"Microphones Axis Defenition\" width=\"400\" height=\"240\">\n",
|
||
"\n",
|
||
"To use the microphone array, you must ensure that the correct sound card driver is installed. The sound card used in this project is the **Scarlett 18i20 3rd Gen**. Below are instructions on how to configure Sounddevice and the necessary drivers on different platforms.\n",
|
||
"\n",
|
||
"### Simulator\n",
|
||
"For the Sounddevice package another simulator has been made. The simulator will return a realistic audio recording, and change the recordings according to the location of the car. But, it contains only 1 recording, so it will not appear as random as the real car. It also does not adjust to your particular beacon settings. Make sure to test on the real car frequently. Use it in combination with the serial simulator to change locations and test like you would on the real car.\n",
|
||
"\n",
|
||
"### Important: Lab rules for the microphone array\n",
|
||
"\n",
|
||
"When working with the microphone array, please follow these rules to ensure smooth operations and avoid disrupting other groups:\n",
|
||
"\n",
|
||
"1. **Do not rearrange the microphone connectors**. The setup is shared between multiple groups, and changing the connections may lead to incorrect results for other teams.\n",
|
||
"2. **Do not touch the volume settings**. If the volume needs adjustment, contact a TA for assistance.\n",
|
||
"3. **Handle the equipment carefully**. The microphone array and associated hardware are sensitive, and mishandling could cause damage.\n",
|
||
"4. **Start and stop on time**. The lab is shared, and other groups have scheduled time slots. Be respectful of their time.\n",
|
||
"\n",
|
||
"Test time is limited. But by using the simulator and preparing a plan of what you want to test during each scheduled slot, there should be enough time to complete the tasks. \n",
|
||
"\n",
|
||
"### Step 1: Initializing the microphone array\n",
|
||
"\n",
|
||
"Before using the microphone array, it must first be initialized. As part of the initialization process, you will need to specify the sampling frequency (`Fs`) that will be used to record the audio. The sampling frequency will vary based on the test field you are working with, and it will be **48 kHz** or **44.1 kHz**.\n",
|
||
"\n",
|
||
"A typical laptop or PC may have multiple audio devices (e.g., built-in microphones, Bluetooth headsets, external sound cards). To ensure that the correct device is used, you can list all available audio devices using Sounddevice and select the appropriate one. Use the following code snippet to list all audio devices recognized by Sounddevice and find the index of the Scarlett 18i20 or any other relevant device:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"sounddevice_handle = sounddevice()\n",
|
||
"for i in range(sounddevice_handle.get_device_count()):\n",
|
||
" device_info = sounddevice_handle.get_device_info_by_index(i)\n",
|
||
" print(i, device_info['name'])\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Once you have identified the index of the microphone array device from the list, you can initialize it by specifying the device index (`device_index`) and the desired sampling frequency (`Fs`).\n",
|
||
"\n",
|
||
"Here’s how you can open the audio stream using Sounddevice:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Initialize sounddevice again\n",
|
||
"sounddevice_handle = sounddevice()\n",
|
||
"\n",
|
||
"# Specify the device index and sampling frequency\n",
|
||
"device_index = 1 # Replace with the index of your microphone device\n",
|
||
"Fs = 44100 # or 48000, depending on the field setup. Try to use 48000 when making your own recordings.\n",
|
||
"\n",
|
||
"# Open the audio stream with 5 channels, 16-bit audio format (paInt16), and the specified sample rate\n",
|
||
"stream = sounddevice_handle.open(input_device_index=device_index,\n",
|
||
" channels=5,\n",
|
||
" format='float32',\n",
|
||
" rate=Fs,\n",
|
||
" input=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 2: Recording audio data\n",
|
||
"\n",
|
||
"To make a recording with the microphone array, you must specify the **length of the recording** as the number of **audio frames** to capture. Each audio frame consists of samples from all 5 microphones. Given that we are using 16-bit audio (2 bytes per sample), each frame will contain **10 bytes** (5 microphones × 2 bytes per sample).\n",
|
||
"\n",
|
||
"Thus, recording **N frames** will produce **10N bytes** of data. Note: The simulator returns a fixed length recording at 44.1 kHz. The real car will return a recording of the length you specify.\n",
|
||
"\n",
|
||
"The following command records `N` frames from the microphone array and stores the result as a numpy array:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"Fs = 44100 # Sampling frequency\n",
|
||
"N = 2*Fs # 2 seconds of audio data\n",
|
||
"samples,_ = stream.read(N)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"At this point, the microphone data is **interleaved**. This means that the first value (`data[0]`) corresponds to the first sample of microphone 0, the second value (`data[1]`) corresponds to the first sample of microphone 1, and so on. For example, `data[5]` contains the second sample of microphone 0, and the pattern continues. To visualize the interleaving of the data, refer to the table below:\n",
|
||
"\n",
|
||
"| data[0] | data[1] | data[2] | data[3] | data[4] | data[5] | data[6] | data[7] | ... |\n",
|
||
"|---------|---------|---------|---------|---------|---------|---------|---------|-----|\n",
|
||
"| mic 0 | mic 1 | mic 2 | mic 3 | mic 4 | mic 0 | mic 1 | mic 2 | ... |\n",
|
||
"| frame 0 | frame 0 | frame 0 | frame 0 | frame 0 | frame 1 | frame 1 | frame 1 | ... |\n",
|
||
"\n",
|
||
"#### Deinterleaving the data\n",
|
||
"\n",
|
||
"To work with the data from each microphone independently, the **interleaved data** must be split into separate streams for each microphone. This process is called **deinterleaving**.\n",
|
||
"\n",
|
||
"Write a function to deinterleave the audio data and store the samples from each microphone in a separate numpy array:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n",
|
||
"\n",
|
||
"# TODO: Reshape the data into a matrix with 5 columns (one for each microphone)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Plotting the audio data\n",
|
||
"\n",
|
||
"Once you've extracted the audio data for each microphone, you can plot it using Python. **Matplotlib** is a commonly used module for creating plots. Plot the audio data from each microphone to visualize the sound captured by the microphone array:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"### Student Version ###\n",
|
||
"\n",
|
||
"# TODO: Plot the data for each microphone"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Step 3: Testing the microphone array\n",
|
||
"Below are some experiments you can perform to test the microphone array, and develop the code.\n",
|
||
"\n",
|
||
"**Part 1: Clapping**: With the real microphones connected, have your teammate clap in front of each microphone in turn. Record the audio data and plot the results. Is the order of the microphones what you expected? How does the sound intensity change as you move from one microphone to another?\n",
|
||
"\n",
|
||
"**Part 2: Beacon detection**: Turn on KITT's beacon and record the results. Can you identify where KITT is located just by observing the shift in the recordings? Change the beacons parameters and see how it affects the recordings.\n",
|
||
"\n",
|
||
"**Part 3: Ideal OOK signal**: Compare the waveform of the recording to an ideal OOK of your code. What can you see, and what do you infer from this? Are some beacon signals better than others? How can you find a good beacon signal? (This point is revisited in Module 3.)\n",
|
||
"\n",
|
||
"**Part 4: Reference recording**: Make some recordings of the beacon at different locations. These recordings will be useful to your teammates working on the localization algorithm. Similarly, make a recording of a single pulse from the beacon close to one of the microphones. Cut out the pulse and save it separately.\n",
|
||
"\n",
|
||
"**Part 5: KITT class**: Add a method to read the microphones to your KITT class in 'Student Code' files. The method should make a stream, turn on the beacon, start the recording, stop the recording, and turn off the beacon. You can choose to return the recording as a result, or store it internally inside the KITT class. Make sure to test your code."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"*Bonus Tasks - Optional*\n",
|
||
"\n",
|
||
"- See if you can automate selecting the correct sounddevice device index. The correct device index changes from one computer to another and can sometimes even change on the same computer after a reboot. So, it is worth your time to make a program that can automatically select the right device index.\n",
|
||
"- Implement start-up sanity checks: some process which you can run after you arrive at the test field, so that you can quickly check the microphone connections and audio levels.\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Mid-term assessment 2.2 and report\n",
|
||
"\n",
|
||
"After you finish this assignment, and ultimo in week 4, showcase the functionality of your script to your\n",
|
||
"assigned TA. After you pass this assessment, you are ready to document your results in your midterm\n",
|
||
"report.\n",
|
||
"\n",
|
||
"For this Module, you would include a chapter that covers the above tasks (using independently-readable\n",
|
||
"text, i.e., don’t refer to “Task 1”). Include plots; for each plot it should be clear how the plot was made\n",
|
||
"(i.e., the corresponding experimental set-up), and you have to describe what is seen in the plot before\n",
|
||
"you discuss results and derive any conclusions. Be sure to answer the questions posed along with the\n",
|
||
"plots (using independently-readable text).\n",
|
||
"\n",
|
||
"Include the corresponding code in an Appendix. Remember to document your code, using comments\n",
|
||
"to define input/output variables of functions and to explain the logic and any modifications made. Your\n",
|
||
"completed script will be crucial for the upcoming challenges, contributing to the overall autonomous\n",
|
||
"driving system.\n",
|
||
"\n",
|
||
"This concludes the mid-term assignments related to communication with KITT. After the mid-term, you\n",
|
||
"must integrate this module with the localization module created by your colleagues. Take into account\n",
|
||
"that integrating is often harder than originally anticipated, e.g. your code has to run in parallel, and you\n",
|
||
"have to worry about timing aspects. Hopefully, using the KITT class will provide you with a sturdy and\n",
|
||
"flexible framework to continue your work towards the final challenge\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## FAQ\n",
|
||
"\n",
|
||
"**What is the beam angle ?**\n",
|
||
"\n",
|
||
"The beam angle of a sensor refers to how wide the sensor's detection area is. It determines how much space the sensor can cover when it sends out signals (like sound or light) to detect objects.\n",
|
||
"\n",
|
||
"To determine the beam angle of ultrasonic sensors mounted in front of the car, you have multiple options: \n",
|
||
"\n",
|
||
"1. **Check the sensor datasheet**: The easiest way or at least a way to get some idea to determine the beam angle is to refer to the manufacturer's datasheet for your specific ultrasonic sensor. The datasheet will typically provide the beam angle, often around 15 to 30 degrees for common ultrasonic sensors. But keep in mind that is for a single sensor and not the current set up! Also, the 'reach' of the sensor is angle dependent: straight ahead, it can see several meters, but at an angle, perhaps just half a meter. \n",
|
||
"\n",
|
||
"2. **Experimental Determination for KITT**:\n",
|
||
" - **Measure detection width**: Place a flat object (like a wall) at a fixed distance in front of the sensor (e.g., 1 meter).\n",
|
||
" - **Move the object**: Move the object left and right to determine the points where the sensor stops detecting the object.\n",
|
||
" - **Calculate the angle**: Measure the distance between these two points (detection width) and the distance from the sensor to the object. You may use the following formula:\n",
|
||
"\n",
|
||
" \n",
|
||
" $$\n",
|
||
" \\text{Beam Angle} = 2 \\times \\arctan\\left(\\frac{\\text{Detection Width}/2}{\\text{Distance to Object}}\\right)$$\n",
|
||
" \n",
|
||
" - This calculation will give you the beam angle in degrees.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**I see random numbers from sensors for large distances. Is my sensor damaged ?**\n",
|
||
"\n",
|
||
"During experiments, you may occasionally receive random or unexpected data from the sensors. This can occur not only when the sensors are operating outside their effective range but also at times when they are within range. Several factors (consider what they might be?) can cause ultrasonic sensors to produce inaccurate readings. Additionally, since there are two sensors—one on the left and one on the right—they might produce different, completly different readings.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Are the ultrasonic sensor measurements for the left and the right side done at exactly the same time ?**\n",
|
||
"\n",
|
||
"If you closely observe the blinking of the small LEDs on the ultrasonic board on the car, you might notice that they turn on and off alternatingly. This indicates a slight time difference in the sensor measurements. This delay is also noticeable and can be measured using a moving car. (The reason to operate the two sensors alternatingly is that otherwise they might interfere on each other.)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.13.3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|