Study guide
4.1 Character Encodings and String Immutability
OpenEDG PCAP — Certified Associate in Python Programming Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: The PCAP-31-03 Exam Blueprint and Test-Day Logistics
1.1 The PCAP-31-03 Blueprint, Scoring, and Exam-Day Logistics
2 Chapter 2: Modules, Packages, and the Standard Library
2.1 Module Imports and Namespace Mechanics 2.2 Module Construction and Execution 2.3 Package Hierarchy and Navigation 2.4 Standard Library Modules and Pip
3 Chapter 3: Exceptions and Error Handling
3.1 Exception Control Flow and Syntax 3.2 The Built-in Exception Hierarchy 3.3 Raising Exceptions and Assertions 3.4 Custom Exceptions and Exception Chaining
4 Chapter 4: Strings, Encodings, and Text Processing
4.1 Character Encodings and String Immutability 4.2 String Indexing, Slicing, and Comparison 4.3 String Inspection and Classification Methods 4.4 String Transformation and Formatting Methods 4.5 Iterating, Sorting, and Comparing Strings Against Other Types
5 Chapter 5: Object-Oriented Programming Fundamentals
5.1 Classes, Instances, and Attributes 5.2 Encapsulation, Privacy, and Name Mangling 5.3 Dunder Methods and Object Lifecycle 5.4 Discovering Class Structure: __name__, __module__, __bases__, and Identity
6 Chapter 6: Advanced OOP, Inheritance, and Metaprogramming
6.1 Single Inheritance, Method Overriding, and Polymorphism 6.2 Multiple Inheritance and Method Resolution Order (MRO) 6.3 Class Methods, Static Methods, and Properties
7 Chapter 7: Functional Python, Comprehensions, and Closures
7.1 List, Dictionary, and Set Comprehensions 7.2 Lambda Expressions and Higher-Order Functions 7.3 Variable Scopes, the LEGB Rule, and Closures
8 Chapter 8: Iterators, Generators, and File I/O
8.1 Iterators and the Iteration Protocol 8.2 Generator Functions and Generator Expressions 8.3 File Input/Output and Context Managers 8.4 Predefined Streams, Handles vs. Streams, and errno Diagnostics
4.1 Character Encodings and String Immutability
Key Takeaways Explain This Topic AI
Python 3 strictly separates human-readable text ( str , Unicode code points) from raw binary data ( bytes , 8-bit octets between 0 and 255).
Character sets evolved from 7-bit ASCII (0–127) to 8-bit ISO-8859-1 (0–255) to Unicode (universal code points mapped via UTF-8, UTF-16, and UTF-32).
The built-in ord() function maps a single 1-character string to its integer Unicode code point, while chr() performs the exact inverse mapping.
Strings are serialized to bytes via str.encode(encoding) and deserialized back via bytes.decode(encoding) .
Python strings are strictly immutable; any modification or concatenation generates a new object in memory with a distinct memory address ( id() ).
Last updated: August 2026
Character Encodings and String Immutability
Understanding how Python handles text at the bit and byte level is a cornerstone of the OpenEDG PCAP-31-03 certification. Python 3 introduced a rigorous model that cleanly separates human-readable characters from raw binary sequences.
1. Text vs. Bytes in Python 3
In Python 2, strings were handled as raw byte sequences by default, which led to frequent encoding errors when dealing with international character sets. Python 3 resolved this by creating two distinct, non-interchangeable sequence types:
str (Text Strings): An immutable sequence of abstract Unicode code points . A str object represents human characters, letters, symbols, and emojis independently of how they are stored as physical bytes in memory.
bytes (Byte Sequences): An immutable sequence of raw 8-bit octets (integers in the range 0 through 255 ). Byte literals use the b prefix (e.g., b'Hello' ). # Creating text and byte objects
- text_data = "Python π"
- byte_data = b"Python"
- print(type(text_data)) # <class 'str'>
- print(type(byte_data)) # <class 'bytes'>
- # Indexing behavior differences
- print(text_data[0]) # 'P' (1-character str)
- print(byte_data[0]) # 80 (integer representing the ASCII value of 'P')
- The Strict Boundary Rule
In Python 3, str and bytes can never be implicitly concatenated or compared for ordered relationships. Attempting to combine them raises a TypeError : # This raises TypeError: can only concatenate str (not "bytes") to str
- result = "Version: " + b"3.10"
- 2. History and Evolution of Character Sets
To understand encoding in Python, it helps to review how computers represent human language numerically. +-------------------------------------------------------------------------+
- | UNICODE |
- | (U+0000 to U+10FFFF: 1,114,112 points) |
- | +-------------------------------------------------------------------+ |
- | | ISO-8859-1 (Latin-1) | |
- | | (0 to 255: 8-bit) | |
- | | +-------------------------------------------------------------+ | |
- | | | ASCII | | |
- | | | (0 to 127: 7-bit) | | |
- | | +-------------------------------------------------------------+ | |
- | +-------------------------------------------------------------------+ |
- +-------------------------------------------------------------------------+
- ASCII (1963)
Structure: 7-bit encoding defining 128 characters (code values 0 through 127 ).
Contents: English uppercase letters ( 65 – 90 for 'A' – 'Z' ), lowercase letters ( 97 – 122 for 'a' – 'z' ), decimal digits ( 48 – 57 for '0' – '9' ), punctuation marks, and non-printable control codes ( 10 for newline \n , 9 for tab \t , 0 for null).
Latin-1 / ISO-8859-1 (1987)
Structure: 8-bit encoding defining 256 characters ( 0 through 255 ).
Contents: The first 128 positions are identical to ASCII, while positions 128 through 255 add accented characters, umlauts, and currency symbols common in Western European languages (e.g., 'é' , 'ñ' , 'ü' ).
The Unicode Standard
Unicode assigns every character across all human languages, historical scripts, and symbol sets a unique numerical identifier called a code point . Unicode code points are conventionally written in hexadecimal notation with the prefix U+ (for example, U+0041 for 'A' , U+03C0 for the Greek letter 'π' , and U+1F600 for the emoji '😀' ). Unicode defines 1,114,112 code points ( U+0000 to U+10FFFF ).
Unicode Transformation Formats (UTF)
Unicode defines the abstract code points, but an encoding determines how those numbers are translated into physical bytes:
- Encoding Byte Width per Character ASCII Compatibility Primary Use Case
- UTF-8 Variable (1 to 4 bytes) 100% backward compatible (1 byte for 0–127) Web, Linux, default in Python 3
- UTF-16 Variable (2 or 4 bytes) Incompatible with 7-bit ASCII Windows internal, Java runtime
UTF-32 Fixed (4 bytes per char) Incompatible with 7-bit ASCII Memory buffers where constant-time indexing is needed
3. Built-in Code Point Functions: ord() and chr()
Python provides two essential built-in functions for converting between characters and their numerical code points: # ord(c): Character -> Integer Code Point
- print(ord('A')) # 65 (ASCII / Unicode)
- print(ord('a')) # 97
- print(ord('0')) # 48
- print(ord('€')) # 8364 (Euro sign)
- print(ord('π')) # 960 (Greek small letter pi)
- # chr(i): Integer Code Point -> Character
- print(chr(65)) # 'A'
- print(chr(97)) # 'a'
- print(chr(8364)) # '€'
- print(chr(960)) # 'π'
- Inverse Invariant Property
For any valid single-character string c and integer code point i : chr ( ord ( c ) ) = = c \text{chr}(\text{ord}(c)) == c chr ( ord ( c )) == c ord ( chr ( i ) ) = = i \text{ord}(\text{chr}(i)) == i ord ( chr ( i )) == i
Exception Traps
ord() accepts only a single character string (length exactly 1). Passing an empty string or multiple characters raises a TypeError : ord("") # TypeError: ord() expected a character, but string of length 0 found
ord("AB") # TypeError: ord() expected a character, but string of length 2 found
chr() accepts integers in the range 0 through 0x10FFFF ( 1,114,111 ). Passing a value outside this range raises a ValueError : chr(-1) # ValueError: chr() arg not in range(0x110000)
- chr(1_114_112) # ValueError: chr() arg not in range(0x110000)
- 4. String Encoding and Decoding Operations
Converting between str (characters) and bytes (octets) requires explicit encoding and decoding: str.encode(encoding="utf-8")
- str =============================> bytes
- (Text) <============================= (Binary)
- bytes.decode(encoding="utf-8") # 1. Encoding: str -> bytes
- message = "Café"
- utf8_bytes = message.encode("utf-8")
- latin1_bytes = message.encode("latin-1")
- print(utf8_bytes) # b'Caf\xc3\xa9' (5 bytes: 'C', 'a', 'f' are 1 byte each; 'é' is 2 bytes)
- print(latin1_bytes) # b'Caf\xe9' (4 bytes: 'é' is 1 byte in Latin-1)
- print(len(utf8_bytes)) # 5
- print(len(latin1_bytes)) # 4
- # 2. Decoding: bytes -> str
- recovered_text = utf8_bytes.decode("utf-8")
- print(recovered_text) # "Café"
- Error Handling Strategies
When encoding or decoding encounters invalid or unrepresentable characters, the errors parameter dictates behavior:
'strict' (default) : Raises UnicodeEncodeError or UnicodeDecodeError .
'ignore' : Drops unencodable characters or undecodable bytes.
'replace' : Inserts '?' during encoding, or the Unicode replacement character \ufffd ( '' ) during decoding. symbol = "pi: π"
- # ASCII cannot represent 'π'
- # strict mode (default)
- # symbol.encode("ascii") -> UnicodeEncodeError
- print(symbol.encode("ascii", errors="ignore")) # b'pi: '
- print(symbol.encode("ascii", errors="replace")) # b'pi: ?'
- 5. String Immutability and the Python Memory Model
In Python, strings are immutable sequence types . Once a str object is allocated in memory, its contents can never be modified in-place. language = "Python"
# Attempting in-place modification raises TypeError
try:
language[0] = "J"
except TypeError as err:
- print(f"Error: {err}")
- # Output: Error: 'str' object does not support item assignment
- Variable Rebinding vs. In-Place Mutation
When you perform operations like concatenation or calling string transformation methods, Python allocates a brand-new string object in memory and rebinds the variable name to point to the new address: text = "Hello"
print(id(text)) # Memory address, e.g., 1402384920384
- text += " World" # Augmented assignment
- print(id(text)) # Different memory address! A new str object was allocated
Performance Consideration: In performance-critical loops, repeatedly concatenating strings using += creates O ( N 2 ) O(N^2) O ( N 2 ) time complexity due to repeated memory allocations and copying. The idiomatic Python approach is accumulating substrings in a list and using str.join() .
String Interning
CPython optimizes memory usage by interning (caching in a lookup table) short strings, ASCII identifiers, and string literals. For interned strings, a is b evaluates to True because both references point to the exact same memory address. However, you should always use == for value equality comparisons, as dynamic runtime strings may not be interned.
6. Escape Characters and Raw Strings
Escape sequences allow you to embed non-printable control codes, quotes, and arbitrary Unicode characters inside string literals using the backslash \ character.
Common Escape Sequences
- Sequence Meaning Hex / ASCII Value
- \n Linefeed / Newline 0x0A (10)
- \t Horizontal Tab 0x09 (9)
- \\ Literal Backslash 0x5C (92)
- \' Single Quote 0x27 (39)
- \" Double Quote 0x22 (34)
- \r Carriage Return 0x0D (13)
- \xHH 8-bit character by 2-digit hex value e.g., \x41 is 'A'
- \uHHHH 16-bit Unicode character by 4-digit hex e.g., \u03c0 is 'π'
- \U00HHHHHH 32-bit Unicode character by 8-digit hex e.g., \U0001F600 is '😀'
- Raw String Literals ( r"..." )
Prefixing a string literal with r or R creates a raw string . In a raw string, backslashes are treated as literal characters rather than escape triggers: # Standard string: \n is converted to a newline (length is 1)
- normal_str = "\n"
- print(len(normal_str)) # 1
- # Raw string: \ and n are two separate literal characters (length is 2)
- raw_str = r"\n"
- print(len(raw_str)) # 2
- print(raw_str) # \n
- # Windows file path use case
- path = r"C:\new_folder\test.py"
- print(path) # C:\new_folder\test.py
Exam Trap: A raw string literal cannot end with an odd number of backslashes (e.g., r"C:\" causes a SyntaxError ), because the trailing backslash escapes the closing quote.