Lexical analysis

A Python program is read by a parser. Input to the parser is a stream of tokens, generated by the lexical analyzer. This chapter describes how the lexical analyzer breaks a file into tokens.

• Line structure
  • Comments
  • Explicit line joining
  • Implicit line joining
  • Blank lines
  • Indentation
• Other tokens
• Identifiers
  • Keywords
• Literals
  • String literals
  • Numeric literals
• Operators
• Delimiters