Arduino Serial Communication

Estimated time to complete this lab: 1 hour

This tutorial is adopted from Arduino tutorial for programmer. They use Linux to play with Arduino, you should replace /dev/ttyX to COMx if you use Windows

Objectives

At the end of this self-learning lab, you should be able to:

Send / receive data between your computer and Arduino

Background. Why do we need serial communication?

Arduino has limited processing power (20MHz CPU clock rate) and our PC has limited I/O capability with hardware devices (e.g. few motors support USB directly).
To allow them to work together, we require serial communication.
For example, the Arduino can be responsible for collect sensor readings. It then sends this data back to the PC where data processing is done, which afterwards commands will be sent from the PC to the Arduino to control output devices accordingly.

Prerequisites

Arduino runs on a modified version of C++. But even if you haven’t learnt C++ before, the examples below should still make sense. Still, you are recommended (but not required) to see this tutorial on the m2_wiki about C++ programming afterwards.

Section 1. Basic Data Types

Data types on different platforms may be implemented differently.
- For example, int is 2 bytes long on Arduino, but it is 4 bytes long on x86.
Therefore, it is recommended to explicitly tell the complier the variable length.
The minimum unit of data storage in computer is byte (8 bits).

Common C language data types

Type	Length	Range
uint8_t	8 bits (1 byte)	0 - 255(2^8 -1)
uint16_t	16 bits (2 bytes)	0 - 65535 (2^16 -1)
int8_t	8 bits (1 byte)	-128 - 127 (-2^7-1 to 2^7) 1 bit for the sign
int16_t	16 bits (2 bytes)	-32768 - 32767 (- 2^15 - 1to 2^15)
char (an ASCII character)	8 bit (1 byte)	0 - 255
float (IEEE standard)	32 bits (4 bytes)	3.4E-38 to 3.4E+38

The 'u' stands for unsigned.
C doesn't have native Boolean data type. Use any non-zero value for true, zero for false. You can include <stdbool.h> to use the bool data type to improve readability.

Section 2. Serial communication using Arduino and Serial Monitor

We can send data to the Arduino via USB Serial using serial monitor in the VSCode. Let's try it!
Upload arduino_serial_demo.ino
With the Arduino is plugged into the computer, open serial monitor in VSCode.
Visit https://docs.platformio.org/en/latest/projectconf/section_env_monitor.html to see how to change settings of serial monitor by editing platformio.ini
Try to send the message "abc". You should see some text show up. That is the message sent by the Arduino!
You may notice that the TX and RX LED light up on the Arduino when you click "send". This is because TX and RX indicate serial communication.

`arduino_serial_demo.ino`

int incomingByte = 0; // for incoming serial data

void setup() {
  Serial.begin(9600); // opens serial port, with 9600 baudrate
}

void loop() {
  // send data only when you receive data:
  if (Serial.available() > 0) {
    // read the incoming byte:

    incomingByte = Serial.read();
    // say what you got:
    Serial.print("I received: ");
    Serial.println(incomingByte, DEC);
  }
}

The comments should be self-explanatory. Some further elaboration is given below:

9600 bps baudrate means that the serial communication speed is 9600 bits per second.
Serial.print() essentially writes a c-string to the serial output, which is actually a sequence of char (1 byte).

The meaning of I received 97 in the serial monitor is that 97 is the decimal form of the character a in ASCII. In ASCII, every character is assigned a number from 0 to 127.

ASCII Table

Try it yourself 1

Connect the Arduino with an LED
- Now, modify the above (or write a new) program such that the brightness of the LED is changed (using analogWrite()) based on the serial input that is received.
- After the brightness is changed, the Arduino should send a response message Updated to %d (where %d is replaced by the brightness) via serial.
- Perform serial I/O via the Serial Monitor.
- Hint: You need to use a function that translates c-string to integer (and perhaps vice-versa).

Section 3. Using Python for serial communication with Arduino

Although we can use the Serial Monitor to send and receive data between Arduino and PC, this way of communication is only suitable for fast prototyping, and is not efficient nor flexible in production code (e.g. ROS node).
We can also use the pyserial library in python to perform communication on the serial port. This can allow us to bridge Arduino board and the computer by ourselves.

pySerial

The serial monitor provides a convenient way for showing serial communication between the Arduino boards and the Computer. However, it does not allow the computer to react according to the value received. No worry, you can write your own Serial Monitor using Python! pySerial is a Library which allows python programs to access the serial port. Thus, we can now bridge Arduino board and the computer by ourselves.

If you haven't tried pySerial or want to further familiarize yourself with Python:

LinuxWindows

Python and pySerial can be easily installed through python-pip. Take Ubuntu as an example.
```
sudo apt install python python-pip
pip install pyserial
```

Install Python https://www.python.org/ or install using Windows store
Remember to add python to your PATH

or do it manually https://datatofish.com/add-python-to-windows-path/

Visit https://phoenixnap.com/kb/install-pip-windows to install pip
Use python -m pip install pyserial``` to install pyserial package

Section 4. Pass data in a packet

Data in the serial monitor is sent as text. This makes communication inefficient.
For example, if the integer 123 is sent as a string “123”, 3 bytes (ASCII characters) are sent and the Arduino needs to convert the string back to an integer.

It is better to use an unsigned 1 byte long integer (uint8_t) to send and represent 123 because only 1 byte is consumed.

Consider a robot with 3 servo motors and 3 DC motors controlled using Arduino
To control the position of the servo and the speed of DC motor, PWM control is needed and the range of the parameter of analogWrite() is from 0 to 255.
To control the 3 servo motors, we need 3 uint8_t fields (each using 1 byte) for the PWM value (0-255) encoding the servo angle.
To control the 3 DC servo motors, we need 3 int16_t fields (each using 2 bytes) to represent the speed of DC motor (-255 to 255) where the sign represents the direction of rotation.

The packet will look like this

1 byte	1 byte	1 byte	2 bytes	2 bytes	2 bytes
uint8_t	uint8_t	uint8_t	int16_t	int16_t	int16_t
angle[0]	angle[1]	angle[2]	speed[0]	speed[1]	speed[2]

Since both x86-64 CPU and the MCU used in Arduino (ATMega328p) store data in little-endian order, we will also use little-endian transmission in this lab.

This is the code used to send data to Arduino, you can try it in python interactive mode.

Python:

import serial
import struct

# Connect to Arduino
arduino = serial.Serial("SERIAL PORT NAME", timeout=2)

angles = [0, 30, 60]   # convert python int to uint8_t
speeds = [-255,0,255]  # convert python int to int16_t

packet = struct.pack("<BBBhhh", *angles, *speeds)
# <BBBhhh stands for 3 uint8_t and 3 int16_t in little endian

# You can see what is in packet by
print(packet.hex()) # 001e3c01ff0000ff00 -> 9 bytes

# send the packet to Arduino
arduino.write(packet)

Arduino:

union{
    char bytes[9];
    struct{  // The struct and bytes[] share same memory location
        uint8_t angles[3];
        int16_t speeds[3];
    }unpacked;
}packet;

void readPacket(){
    while (Serial.available() < 9);
            // do nothing and wait if serial buffer doesn't have enough bytes to read
    Serial.readBytes(packet.bytes, 9);
            // read 9 bytes from serial buffer and store them at a
            // union called “packet”
}

void setup() {  // put your setup code here, to run once:
    Serial.begin(9600);
    Serial.setTimeout(2000);
        // give up waiting if nothing can be read in 2s
}

void loop() {  // put your main code here, to run repeatedly:
    int i;
    readPacket(); // read 9 bytes and populate the packet union
    for(int i=0; i<9; i++)
        Serial.write(packet.bytes[i]); // send back the packet
    for(int i=0; i<3; i++)
        Serial.println(packet.unpacked.angles[i]); 
            // send back angle as string
    for(int i=0; i<3; i++)
        Serial.println(packet.unpacked.speeds[i]);
            // send back speeds as string
}

After sending a 9 bytes packet to Arduino, the Arduino will reply with the same packet followed by a string of 3 angles + 3 speeds, print the values of angles and speeds in human readable string. You can read this in python with the following code:

import serial
import struct
import time
# Connect to Arduino
arduino = serial.Serial("COM4", timeout=1)

angles = [0, 30, 60]   # convert python int to uint8_t
speeds = [-255,0,255]  # convert python int to int16_t

packet = struct.pack("<BBBhhh", *angles, *speeds)
# <BBBhhh stands for 3 uint8_t and 3 int16_t in little endian

while True:
  # See what is in packet
    print("sending:", packet.hex()) # 001e3c01ff0000ff00 -> 9 bytes
  # send the packet to Arduino
    arduino.write(packet)
    time.sleep(1) # wait Arduino process and reply
    print("received:")
    print("Original packet:", arduino.read(9).hex())
    print("Values")
    print(arduino.read(100).decode('ascii')) # print out human readable string

Section 5. Packet Framing and Checksum

If you are lucky, the previous example works, but it is not an robust way to transmit data. In reality, these things might happen:

Some bytes are lost during transmission
Some bits are flipped during transmission
Your MCU keep sending data and you don't know which byte is the first byte of the packet

This will lead to misinterpretation of the data, and this can be dangerous, e.g. if you send 0x0001 to command the Arduino to output 1A to the motor but the first byte magically disappears and the Arduino mistakenly treat the first byte of the next command as the second byte of the current command, it will become 0x01?? or if the first byte is flipped it will become 0x8001. This might cause unimaginable consequence. So, we will need a more reliable way to transmit our data.

We can do the followings while framing our data to make it more reliable:

Insert escape byte and flag byte to indicate the start and end of a packet
Insert a byte to indicate how the length of your data if your packet contains variable length data (optional for fixed length packet)
Insert a checksum byte just before the end of a packet

The packet will look like this.

Byte 1	Byte 2	Byte 3	Byte 4	Byte 5	Byte 6	Byte 7
DLE (0x10)	STX (0x02)	LEN (opt.)	DATA	CHECKSUM	DLE (0x10)	ETX (0x03)

The reason why we are inserting DLE before STX and ETX is to reduce the chance of mistakenly treating 0x02 and 0x03 in your packet as the start byte or end byte.

Example

Arduino: Receive command from computer and reply with actual outputs

#include <Arduino.h>

struct Data
{ // The struct and bytes[] share same memory location
  uint8_t angles[3];
  int16_t speeds[3];
};

struct Packet
{
  // send first
  uint16_t start_seq; // 0x0210, 0x10 will be sent first
  uint8_t len;        // length of payload
  struct Data tx_data;
  uint8_t checksum;
  uint16_t end_seq;   // 0x0310, 0x10 will be sent first
  // send last
};

struct Packet tx_packet; // store packet to be sent
struct Data rx_data; // store received data

/*
  Calculate checksum by XOR-ing all the byte where the pointer "data" points to
  @param data starting address of the data
  @param len length of the data
  @return caculated checksum
*/
uint8_t calc_checksum(void *data, uint8_t len)
{
  uint8_t checksum = 0;
  uint8_t *addr;
  for(addr = (uint8_t*)data; addr < (data + len); addr++){
    // xor all the bytes
    checksum ^= *addr; // checksum = checksum xor value stored in addr
  }
  return checksum;
}

/*
  Read packet from serial buffer
  @return whether a packet is received successfully
*/
bool readPacket()
{
  uint8_t payload_length, checksum, rx;
  while(Serial.available() < 15){
    // not enough bytes to read
  }
  char tmp[15];

  if(Serial.read() != 0x10){
    // first byte not DLE, not a valid packet
    return false;
  }

  // first byte is DLE, read next byte
  if(Serial.read() != 0x02){
    // second byte not STX, not a valid packet
    return false;
  }

  // seems to be a valid packet
  payload_length = Serial.read(); // get length of payload

  // can compare payload length or extra packet type byte to decide where to write received data to
  if(payload_length == 9){
    if(Serial.readBytes((uint8_t*) &rx_data, payload_length) != payload_length){
      // cannot receive required length within timeout
      return false;
    }
  }else{
    // invalid data length
    return false;
  }

  checksum = Serial.read();

  if(calc_checksum(&rx_data, payload_length) != checksum){
    // checksum error
    return false;
  }

  if(Serial.read() != 0x10){
    // last 2nd byte not DLE, not a valid packet
    return false;
  }

  // last 2nd byte is DLE, read next byte
  if(Serial.read() != 0x03){
    // last byte not ETX, not a valid packet
    return false;
  }

  // Yeah! a valid packet is received

  return true;
}

void send_packet(){
  tx_packet.len = sizeof(struct Data);

  // mimick actual output
  tx_packet.tx_data.angles[0] = rx_data.angles[0] + 5;
  tx_packet.tx_data.angles[1] = rx_data.angles[1] - 5;
  tx_packet.tx_data.angles[2] = rx_data.angles[2] + 3;
  tx_packet.tx_data.speeds[0] = rx_data.speeds[0] + 15;
  tx_packet.tx_data.speeds[1] = rx_data.speeds[1] + 3;
  tx_packet.tx_data.speeds[2] = rx_data.speeds[2] - 11;

  tx_packet.checksum = calc_checksum(&tx_packet.tx_data, tx_packet.len);
  Serial.write((char*)&tx_packet, sizeof(tx_packet)); // send the packet
}
void setup()
{ // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.setTimeout(5000);  // give up waiting if nothing can be read in 2s

  // init tx packet
  tx_packet.start_seq = 0x0210;
  tx_packet.end_seq = 0x0310;

  while(!Serial){
    // wait until Serial is ready
  }
}

void loop()
{ // put your main code here, to run repeatedly:
  if(readPacket()){
    // valid packet received, pack new data in new packet and send it out
    send_packet();
  }
}

Python: Send command to Arduino and print out actual outputs

import serial
import struct
import time

angles = [0, 30, 60]
speeds = [-255,0,255]
arduino = serial.Serial("COM4", timeout=1, baudrate=9600)
time.sleep(3) # Arduino will be reset when serial port is opened, wait it to boot

def calc_checksum(data):
    calculated_checksum = 0
    for byte in data:
        calculated_checksum ^= byte
    return calculated_checksum

def read_packet():
    '''
    :return received data in the packet if read sucessfully, else return None
    '''
    # check start sequence
    if arduino.read() != b'\x10':
        return None

    if arduino.read() != b'\x02':
        return None

    payload_len = arduino.read()[0]
    if payload_len != 9:
        # could be other type of packet, but not implemented for now
        return None

    # we don't know if it is valid yet
    payload = arduino.read(payload_len)

    checksum = arduino.read()[0]
    if checksum != calc_checksum(payload):
        return None # checksum error

    # check end sequence
    if arduino.read() != b'\x10':
        return None
    if arduino.read() != b'\x03':
        return None    

    # yeah valid packet received
    return payload

def send_packet():
    tx = b'\x10\x02' # start sequence
    tx += struct.pack("<B", 9) # length of data
    packed_data = struct.pack("<BBBhhh", *angles, *speeds)
    tx += packed_data
    tx += struct.pack("<B", calc_checksum(packed_data))
    tx += b'\x10\x03' # end sequence
    print("Sending:", tx.hex())
    arduino.write(tx)

def main():
    timeout = 3
    try:
        # keep sending packet and try to unpack received packet
        while True:
            print("Setting angle outputs", angles) # first 3 unpacked values
            print("Setting speed outputs", speeds) # next 3 unpacked values
            send_packet()
            time.sleep(0.5)

            start_time = time.time()
            payload = None
            while(payload == None and time.time() < (start_time+timeout)):
                payload = read_packet()
            if payload == None:
                print("timeout")
                continue
            # a packet is read successfully
            unpacked = struct.unpack("<BBBhhh", payload)
            print("Actual angle outputs", unpacked[:3]) # first 3 unpacked values
            print("Actual speed outputs", unpacked[3:]) # next 3 unpacked values

            print("\n\n")
    except KeyboardInterrupt:
        exit(0)

main()

Reference

https://eli.thegreenplace.net/2009/08/12/framing-in-serial-communications

Arduino Serial Communication

Background. Why do we need serial communication?

Prerequisites

Section 1. Basic Data Types

Common C language data types

Section 2. Serial communication using Arduino and Serial Monitor

arduino_serial_demo.ino

Section 3. Using Python for serial communication with Arduino

pySerial

Section 4. Pass data in a packet

Section 5. Packet Framing and Checksum

Example

Further reading

Reference

`arduino_serial_demo.ino`