Python Analyser specificities¶
Note
This documentation applies to CAIP >= 8.3.x and com.castsoftware.python >= 1.4.0-beta2.
HowTo¶
Create your own UA extension, then react to the “add_quality_rules” event received from Python extension. With this event, you can send any interpreters of your choice, which will be used by the extension. These interpreters will have “start_XXX” and/or “end_XXX” methods where XXX is the ast node type you are interested in. They can also have also a “finish” method that is called after all python files have been analyzed.
Sample of UA extension code
import cast.analysers.ua
from cast import Event
from cast.analysers import log
from interpreters import Interpreter1, Interpreter2
class CustomPythonDiag(cast.analysers.ua.Extension):
@Event('com.castsoftware.python', 'add_quality_rules')
def add_quality_rules(self, externalQualityRules):
# Add the interpreters
externalQualityRules.add_interpreter(Interpreter1, self.get_plugin())
externalQualityRules.add_interpreter(Interpreter2, self.get_plugin())
...
Sample of a simple interpreter code
class Interpreter:
def __init__(self):
pass
def start_MethodCall(self, ast):
...
def end_MethodCall(self, ast):
...
def start_XXX(self, ast):
...
def end_XXX(self, ast):
...
...
A complete sample is available here: Starter kits.
Python ast node types¶
Here we present the main classes and methods that form the API to exploit the information contained in the abstract syntax tree (AST) of a python file.
Basic elements¶
The AST returned by the Python analyzer is composed of nodes (Node type) and tokens (Token type). Tokens are the most basic elements. All nodes do contain sub-items (other sub-nodes and tokens). In some specific contexts one might not distinguish between nodes and tokens as they are treaded similarly.
All nodes inherit direcly or indirectly from light_parser.Node. Some nodes might inherit from some generic intermediate classes (Statement, BlockStatement, Term) used internally for parsing purposes. The character of the node (expression vs statement) is loosely correlated to the the name of these clases for certain nodes. These intermediate classes do not contribute to the API.
- class light_parser.Token(text=None, type=None)¶
A token with code position.
- get_begin_line()¶
Returns: the ast node begin line Return type: int
- get_begin_column()¶
Returns: the ast node begin column Return type: int
- get_end_line()¶
Returns: the ast node end line Return type: int
- get_end_column()¶
Returns: the ast node end column Return type: int
- get_type()¶
Returns: the name of the token type Return type: str
- get_children()¶
Convenient method to comply with ‘light_parser.Node’ interface.
Returns: empty list
- get_sub_nodes()¶
Convenient method to comply with ‘light_parser.Node’ interface.
Returns: empty list
- __eq__(self, other):
Equality.
Can compare to text directly. (case insensitive).
- class light_parser.Node¶
An AST node
- get_begin_line()¶
Returns: the ast node begin line Return type: int
- get_begin_column()¶
Returns: the ast node begin column Return type: int
- get_end_line()¶
Returns: the ast node end line Return type: int
- get_end_column()¶
Returns: the ast node end column Return type: int
- get_type()¶
Returns: the name of the node Return type: str
- get_type()¶
- get_children()¶
An iterator on tokens and children that skips whitespaces and comments
- get_sub_nodes(_type=None)¶
Return sub AST nodes
if a _type is provided returns only sub nodes of that type.
- print_tree(depth=0)¶
Print as a tree.
The following intermediate classes inherit from Node, but do not contribute to the API.
- class light_parser.Statement¶
- class light_parser.BlockStatement¶
- class light_parser.Term¶
Base classes¶
The classes below conveniently factorize common code among certain type of nodes. These are not explicit nodes of the AST.
- class python_parser.PythonStatement¶
- get_container_block()¶
Access to block, if, for etc... that contains that statement
@return: None for top level statements
- class python_parser.WithDocString¶
- get_docstring(unstripped=False)¶
Return python docstring of element
- get_decorators()¶
Access to decorators of object.
@rtype: list of Decorator
- class python_parser.Class¶
Bases: python_parser.PythonStatement, python_parser.WithDocString
Base class for Python classes
- is_class()¶
Return type: boolean
- get_name()¶
Access to class name
Return type: str
- get_inheritance()¶
Access to inheritance
- class python_parser.While¶
- class python_parser.With¶
Common class for with statement
Statements¶
Definition statements¶
- class python_parser.ClassBlock¶
Bases: python_parser.Class, light_parser.BlockStatement
Class definition containing an indented statement block
- class python_parser.ClassOneLine¶
Bases: python_parser.Class, light_parser.Statement
Class definition in one line
- class python_parser.FunctionBlock¶
Bases: python_parser.Function, light_parser.BlockStatement
Function definition containing an indented statement block
- class python_parser.FunctionOneLine¶
Bases: python_parser.Function, light_parser.Statement
Function definition in one line
- class python_parser.Global¶
- class python_parser.NonLocal¶
Simple statements¶
- class python_parser.ExpressionStatement¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement
- get_expression()¶
- is_expression_statement()¶
- class python_parser.Import¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement
- is_import()¶
- get_module_references()¶
Access to the imported module
Returns: ‘a.b’ for import a.b
- get_imported_elements()¶
from ... import <imported elements>
- class python_parser.Return¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Yield¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.YieldFrom¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Await¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Delete¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Break¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement
- class python_parser.Continue¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement
- class python_parser.Raise¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Print¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Exec¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Assert¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression
- class python_parser.Pass¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement
- class python_parser.EllipsisObject¶
Bases: light_parser.Statement, python_parser.PythonSimpleStatement
Block statements¶
Statements containing other statements.
- class python_parser.ForBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.For
- class python_parser.ForOneLine¶
Bases: light_parser.Statement, python_parser.For
- class python_parser.WhileBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.While
- class python_parser.WhileOneLine¶
- class python_parser.TryBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.PythonStatement
- class python_parser.TryOneLine¶
Bases: light_parser.Statement, python_parser.PythonStatement
- class python_parser.ExceptBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.PythonStatement, python_parser.WithExpression
- class python_parser.ExceptOneLine¶
Bases: light_parser.Statement, python_parser.PythonStatement
- class python_parser.WithBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.With
A ‘with’ statement of the form:
- with ... as ...:
- stmt stmt ...
- class python_parser.WithOneLine¶
Bases: light_parser.Statement, python_parser.With
- with statement
- with ... as ...: ...; ...;
- get_statements()¶
- class python_parser.FinallyBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.PythonStatement
- class python_parser.FinallyOneLine¶
Bases: light_parser.Statement, python_parser.PythonStatement
- class python_parser.IfThenElseBlock¶
Bases: python_parser.PythonBlockStatement, python_parser.IfThenElse
Classical block
- class python_parser.IfThenElseOneLine¶
Bases: light_parser.Statement, python_parser.IfThenElse
an if can be on one line : if .... : statement1; ... statementN;
- get_statements()¶
Expressions¶
- class python_parser.Identifier¶
Bases: python_parser.Expression
An identifier, e.g. : - a - f
No dots here, see DotAccess
- get_name()¶
- is_identifier()¶
- get_assigned_expression()¶
- get_resolution()¶
One might use this instead of ‘get_resolutions’ when it is highly unlikely to have ambiguity in resolution (or not important).
- class python_parser.BinaryOperation¶
Bases: python_parser.Expression
A binary operation of the form:
left_expression op right_expression- get_left_expression()¶
- get_right_expression()¶
- is_binary_operation()¶
Return type: Boolean
- get_operator()¶
- is_assignment()¶
- class python_parser.Assignment¶
Bases: python_parser.BinaryOperation
A assignment (includes augmented assignments)
- augassign = ['=', '+=', '-=', '*=', '/=', '@=', '%=', '&=', '|=', '^=', '<<=', '>>=', '**=', '//=']¶
- get_operator()¶
Return type: light_parser.Token
- is_assignment()¶
- get_assigned_variables()¶
Used for handling assignment destructuring
- Limitation:
- Unwrapping of parentheses not handled
- get_assignee()¶
It returns the leftmost element in ‘get_assigned_variables’.
- Use this for simple variable assignments of type
- a = expression
Returns: Leftmost variable of the assignment Return type: python_parser.Node (Identifier, ...)
- class python_parser.Decorators¶
Bases: light_parser.Statement
A suite of decorators. We use this for matching the block of decorators.
- class python_parser.Unpacking¶
Bases: python_parser.Expression
Unpacking (unary) operation like in *args, and **kwargs.
The name of the operator can be ‘*’ or ‘**’.
- get_name()¶
- class python_parser.Constant¶
Bases: python_parser.Expression
- get_value()¶
- is_constant()¶
- is_constant_string()¶
- is_fstring()¶
- get_string_value()¶
- class python_parser.DotAccess¶
Bases: python_parser.Expression
exp . accname
- get_expression()¶
Returns: exp Return type: Expression
- get_identifier()¶
Returns: accname Return type: Identifier
- get_name()¶
Return type: str
- is_dot_access()¶
- class python_parser.ArrayAccess¶
Bases: python_parser.Expression
exp [...]
- get_array_expression()¶
- get_indice()¶
- class python_parser.MethodCall¶
Bases: python_parser.Expression
exp (...)
- get_method()¶
Return the expression without the parenthesis
f() -> f a.b() -> a.b ...
- get_parameters()¶
- is_method_call()¶
- get_argument(position, keyword)¶
Returns positional/keyword argument
If found a match with positional argument, the keyword argument is neglected.
- Limitations
- Optional keyword arguments with no defined position can be retrieved by keyword. However combinations with optional (non-keyword) arguments are not tested.
- is_resolved_to(qname)¶
- class python_parser.Array¶
Bases: python_parser.Expression
An array expression :
[..., ..., ...]
- get_values()¶
- class python_parser.ComprehensionLoop¶
Bases: python_parser.Expression
[ ... for ... in ... ...] ( ... for ... in ... ...)
- get_expression()¶
return first expression
- get_comprehension_for()¶
Get the for part
- class python_parser.ComprehensionFor¶
Bases: light_parser.Node, python_parser.For
The sub part of a comprehension loop
- class python_parser.Map¶
Bases: python_parser.Expression, python_parser.WithResolution
A map expression:
{...:..., ...:..., ...:...}
- get_values()¶
- is_dict_like()¶
- class python_parser.Lambda¶
Bases: python_parser.Expression
- class python_parser.LambdaParameters¶
Bases: python_parser.Expression
- class python_parser.Unary¶
Bases: python_parser.Expression
Unary expression
- class python_parser.UnaryNot¶
Bases: python_parser.Unary
not expression
- class python_parser.AdditionExpression¶
Bases: python_parser.BinaryOperation
exp + exp
- is_addition()¶
- class python_parser.StringInterpolation¶
Bases: python_parser.BinaryOperation
Expressions of type exp % exp
- Limitations:
- It over-captures modulo-operator expressions
- is_interpolation()¶
Symbols¶
Symbols closely represent the structure of metamodel objects, among other functionalities. They are also used for resolution purposes for certain nodes.
- class symbols.Symbol(name, parent=None)¶
base class for symbols
- get_parent_symbol()¶
- get_root_symbol()¶
- get_name()¶
Return name
- get_qualified_name()¶
Qualified name
- get_ast()¶
Access to AST of symbol.
- print_tree(depth=0)¶
Print as a tree.
- get_statements()¶
Get the statements of the symbol.
- get_class(name, begin_line=None)¶
Return a class by name. From the classes inside that symbol.
- get_function(name, begin_line=None)¶
Return a function by name From the functions inside that symbol.
- get_begin_line()¶
- get_begin_column()¶
- get_end_line()¶
- get_end_column()¶
- get_violations(property_name)¶
Returns all violations for a given rule.
- add_violation(property_name, ast, *additional_asts)¶
Add a violation for a quality rule.
Parameters: - property_name – fullname of the property
- ast – location of the violation
- set_property(property_name, value)¶
Used to set a generic property on object
mainly used for quality rules
Parameters: - property_name – fullname of the property
- value – value of the property
- is_class()¶
- is_function()¶
- class symbols.Module(path, _file=None, text=None)¶
Bases: symbols.Symbol
A Python file
- metamodel_type = 'CAST_Python_SourceCode'¶
- is_package()¶
True when module is a package.
- get_path()¶
- get_file()¶
- get_qualified_name()¶
Qualified name
- get_statements()¶
Get the statements of the symbol.
- get_main()¶
Return the ‘main’ found in that module
- get_fullname()¶
- find_local_symbols(name, types=[])¶
Override for modules.
- class symbols.Class(name, parent)¶
Bases: symbols.Symbol
A python class
- metamodel_type = 'CAST_Python_Class'¶
- get_inheritance()¶
Inherited classes.
Return type: list of python_parser.Reference
- get_ordered_inheritances()¶
Return the list of all inherited classes ordered by the Python Method Resolution Order (known as MRO, introduced in Python 2.3). Only those resolved classes are returned.
Return type: list of symbols.Class
- find_method(name)¶
Find a method of the class, using inheritance
- find_member(name)¶
- is_callable(method)¶
Says that :param method: is callable on that class.
Handle override.
- is_callable_by_name(name)¶
Says that :param method: is callable on that class.
Handle override.
- get_fullname()¶
- get_members()¶
- get_classes()¶
- get_functions()¶
- is_class()¶
- class symbols.Function(name, parent, return_type=None)¶
Bases: symbols.Symbol
A Python function or method
- metamodel_type = 'CAST_Python_Method'¶
- get_fullname()¶
- is_function()¶
Writing tests¶
Sample of a test
import unittest
import cast.analysers.test
import cast.analysers.ua
class Test(unittest.TestCase):
def test_ok1(self):
analysis = cast.analysers.test.UATestAnalysis('PYTHON') # set Python language
analysis.add_selection('test1')
analysis.add_dependency(r'C:\ProgramData\CAST\CAST\Extensions\com.castsoftware.python.1.4.0-beta2')
analysis.run()
f = analysis.get_object_by_name('f', 'CAST_Python_Function')
self.assertTrue(f)
value = f.get_value('CAST_Python_Rule.AvoidUsingInvalidObjectsInAll')
self.assertEqual('1', value)
if __name__ == '__main__':
unittest.main()
Note
You need to add a dependency by absolute path or relative to the folder containing com.castsoftware.python, for example
analysis.add_dependency(r'C:\ProgramData\CAST\CAST\Extensions\com.castsoftware.python.1.4.0-beta2')
For that you need to have downloaded the extension through CAST ExtensionDowloader: see https://doc.castsoftware.com/display/EXTEND/CAST+Extension+Downloader
Categories and types for quality rules¶
For adding a quality rule use the following categories
- Category CAST_Python_Rule (for artifacts), CAST_Python_ClassRule (only for classes).
Quality rule scopes¶
Available basic quality rule scopes are :
1021000 | Callable Python artifacts (includes functions methods and modules) |
1021008 | Python Functions (includes functions and methods) |
1021011 | Python Classes |
1021012 | Python Modules |
Python technologies¶
Values for technology to be used in IMPLTechnologies.xml for filter value
Python | 1021000 |