Python Backport Compiler Utilities¶
Utility library for the Python bpc
backport compiler.
Currently, the three individual tools (f2format
, poseur
,
walrus
) depend on this repo. The bpc
compiler is a
work in progress.
Module contents¶
-
exception
bpc_utils.
BPCSyntaxError
[source]¶ Bases:
SyntaxError
Syntax error detected when parsing code.
-
class
bpc_utils.
BaseContext
(node, config, *, indent_level=0, raw=False)[source]¶ Bases:
abc.ABC
Abstract base class for general conversion context.
-
__iadd__
(code)[source]¶ Support of the
+=
operator.If
self._prefix_or_suffix
isTrue
, then thecode
will be appended toself._prefix
; else it will be appended toself._suffix
.- Parameters
code (str) – code string
- Returns
self
- Return type
-
_process
(node)[source]¶ Recursively process parso AST.
All processing methods for a specific
node
type are defined as_process_{type}
. This method first checks if such processing method exists. If so, it will call such method on thenode
; otherwise it will traverse through all children ofnode
, and perform the same logic on each child.- Parameters
node (parso.tree.NodeOrLeaf) – parso AST
-
_walk
(node)[source]¶ Start traversing the AST module.
The method traverses through all children of
node
. It first checks if such child has the target expression. If so, it will toggleself._prefix_or_suffix
(set toFalse
) and save the last previous child asself._node_before_expr
. Then it processes the child withself._process
.- Parameters
node (parso.tree.NodeOrLeaf) – parso AST
-
static
extract_whitespaces
(node)[source]¶ Extract preceding and succeeding whitespaces from the node given.
-
abstract
has_expr
(node)[source]¶ Check if node has the target expression.
- Parameters
node (parso.tree.NodeOrLeaf) – parso AST
- Returns
if
node
has the target expression- Return type
-
static
missing_newlines
(prefix, suffix, expected, linesep)[source]¶ Count missing blank lines for code insertion given surrounding code.
-
static
split_comments
(code, linesep)[source]¶ Separates prefixing comments from code.
This method separates prefixing comments and suffixing code. It is rather useful when inserting code might break shebang and encoding cookies (PEP 263), etc.
-
property
string
¶ Returns conversion buffer (
self._buffer
).
-
-
class
bpc_utils.
Config
(**kwargs)[source]¶ Bases:
collections.abc.MutableMapping
Configuration namespace.
This class is inspired from
argparse.Namespace
for storing internal attributes and/or configuration variables.
-
class
bpc_utils.
UUID4Generator
(dash=True)[source]¶ Bases:
object
UUID 4 generator wrapper to prevent UUID collisions.
-
bpc_utils.
TaskLock
()[source]¶ Function that returns a lock for possibly concurrent tasks.
- Returns
a lock for possibly concurrent tasks
- Return type
Union[contextlib.nullcontext, multiprocessing.synchronize.Lock]
-
bpc_utils.
archive_files
(files, archive_dir)[source]¶ Archive the list of files into a tar file.
- Parameters
files (List[str]) – a list of files to be archived (should be absolute path)
archive_dir (os.PathLike) – the directory to save the archive
- Returns
path to the generated tar archive
- Return type
-
bpc_utils.
detect_encoding
(code)[source]¶ Detect encoding of Python source code as specified in PEP 263.
-
bpc_utils.
detect_files
(files)[source]¶ Get a list of Python files to be processed according to user input.
This will perform glob expansion on Windows, make all paths absolute, resolve symbolic links and remove duplicates.
- Parameters
files (List[str]) – a list of files and directories to process (usually provided by users on command-line)
- Returns
a list of Python files to be processed
- Return type
List[str]
See also
See
expand_glob_iter()
for more information.
-
bpc_utils.
detect_indentation
(code)[source]¶ Detect indentation of Python source code.
- Parameters
code (Union[str, bytes, TextIO, parso.tree.NodeOrLeaf]) – the code to detect indentation
- Returns
the detected indentation sequence
- Return type
Notes
In case of mixed indentation, try voting by the number of occurrences of each indentation value (spaces and tabs).
When there is a tie between spaces and tabs, prefer 4 spaces for PEP 8.
-
bpc_utils.
detect_linesep
(code)[source]¶ Detect linesep of Python source code.
- Parameters
code (Union[str, bytes, TextIO, parso.tree.NodeOrLeaf]) – the code to detect linesep
- Returns
the detected linesep (one of
'\n'
,'\r\n'
and'\r'
)- Return type
Literal[‘\n’, ‘\r\n’, ‘\r’]
Notes
In case of mixed linesep, try voting by the number of occurrences of each linesep value.
When there is a tie, prefer
LF
toCRLF
, preferCRLF
toCR
.
-
bpc_utils.
first_non_none
(*args)[source]¶ Return the first non-
None
value from a list of values.- Parameters
*args –
variable length argument list
If one positional argument is provided, it should be an iterable of the values.
If two or more positional arguments are provided, then the value list is the positional argument list.
- Returns
the first non-
None
value, if all values areNone
or sequence is empty, returnNone
- Return type
Any
- Raises
TypeError – if no arguments provided
-
bpc_utils.
first_truthy
(*args)[source]¶ Return the first truthy value from a list of values.
- Parameters
*args –
variable length argument list
If one positional argument is provided, it should be an iterable of the values.
If two or more positional arguments are provided, then the value list is the positional argument list.
- Returns
the first truthy value, if no truthy values found or sequence is empty, return
None
- Return type
Any
- Raises
TypeError – if no arguments provided
-
bpc_utils.
get_parso_grammar_versions
(minimum=None)[source]¶ Get Python versions that parso supports to parse grammar.
- Parameters
minimum (str) – filter result by this minimum version
- Returns
a list of Python versions that parso supports to parse grammar
- Return type
List[str]
- Raises
ValueError – if
minimum
is invalid
-
bpc_utils.
map_tasks
(func, iterable, posargs=None, kwargs=None, *, processes=None, chunksize=None)[source]¶ Execute tasks in parallel if
multiprocessing
is available, otherwise execute them sequentially.- Parameters
func (Callable) – the task function to execute
iterable (Iterable[Any]) – the items to process
posargs (Optional[Iterable[Any]]) – additional positional arguments to pass to
func
kwargs (Optional[Mapping[str, Any]]) – keyword arguments to pass to
func
processes (Optional[int]) – the number of worker processes (default: auto determine)
chunksize (Optional[int]) – chunk size for multiprocessing
- Returns
the return values of the task function applied on the input items and additional arguments
- Return type
List[Any]
-
bpc_utils.
parse_boolean_state
(s)[source]¶ Parse a boolean state from a string representation.
These values are regarded as
True
:'1'
,'yes'
,'y'
,'true'
,'on'
These values are regarded as
False
:'0'
,'no'
,'n'
,'false'
,'off'
Value matching is case insensitive.
- Parameters
s (Optional[str]) – string representation of a boolean state
- Returns
- Return type
Optional[bool]
- Raises
ValueError – if
s
is an invalid boolean state value
See also
See
_boolean_state_lookup
for default lookup mapping values.
-
bpc_utils.
parse_indentation
(s)[source]¶ Parse indentation from a string representation.
If an integer or a string of positive integer
n
is specified, then indentation isn
spaces.If
't'
or'tab'
is specified, then indentation is tab.- If
'\t'
(the tab character itself) or a string consisting only of the space character (U+0020) is specified, it is returned directly.
- If
Value matching is case insensitive.
-
bpc_utils.
parse_linesep
(s)[source]¶ Parse linesep from a string representation.
These values are regarded as
'\n'
:'\n'
,'lf'
These values are regarded as
'\r\n'
:'\r\n'
,'crlf'
These values are regarded as
'\r'
:'\r'
,'cr'
Value matching is case insensitive.
- Parameters
s (Optional[str]) – string representation of linesep
- Returns
the parsed linesep result, return
None
if input isNone
or empty string- Return type
Optional[Literal[‘\n’, ‘\r\n’, ‘\r’]]
- Raises
ValueError – if
s
is an invalid linesep value
See also
See
_linesep_lookup
for default lookup mapping values.
-
bpc_utils.
parse_positive_integer
(s)[source]¶ Parse a positive integer from a string representation.
-
bpc_utils.
parso_parse
(code, filename=None, *, version=None)[source]¶ Parse Python source code with parso.
- Parameters
- Returns
parso AST
- Return type
parso.python.tree.Module
- Raises
BPCSyntaxError – when source code contains syntax errors
-
bpc_utils.
recover_files
(archive_file)[source]¶ Recover files from a tar archive.
- Parameters
archive_file (os.PathLike) – path to the tar archive file
Internal utilities¶
-
class
bpc_utils.
MakeTextIO
(obj)[source]¶ Bases:
object
Context wrapper class to handle
str
and file objects together.- Variables
-
bpc_utils.
_mp_map_wrapper
(args)[source]¶ Map wrapper function for
multiprocessing
.- Parameters
args (Tuple[Callable, Iterable[Any], Mapping[str, Any]]) – the function to execute, the positional arguments and the keyword arguments packed into a tuple
- Returns
the function execution result
- Return type
Any
-
bpc_utils.
_mp_init_lock
(lock)[source]¶ Initialize lock for
multiprocessing
.- Parameters
lock (multiprocessing.synchronize.Lock) – the lock to be shared among tasks
-
bpc_utils.
expand_glob_iter
(pathname)¶ Wrapper function to perform glob expansion.
-
bpc_utils.
_boolean_state_lookup
= {'0': False, '1': True, 'false': False, 'n': False, 'no': False, 'off': False, 'on': True, 'true': True, 'y': True, 'yes': True}¶ A mapping from string representation to boolean states. The values are used for
parse_boolean_state()
.
-
bpc_utils.
_linesep_lookup
= {'\n': '\n', '\r': '\r', '\r\n': '\r\n', 'cr': '\r', 'crlf': '\r\n', 'lf': '\n'}¶ A mapping from string representation to linesep. The values are used for
parse_linesep()
.
-
bpc_utils.
mp
: Optional[ModuleType] = <module 'multiprocessing'>¶ An alias of the Python builtin
multiprocessing
module if available.
-
bpc_utils.
task_lock
: Union[contextlib.nullcontext, multiprocessing.synchronize.Lock]¶ A lock for possibly concurrent tasks.