Python Analyser specificities

Note

This documentation applies to CAIP >= 8.3.x and com.castsoftware.python >= 1.4.0-beta2.

HowTo

Create your own UA extension, then react to the “add_quality_rules” event received from Python extension. With this event, you can send any interpreters of your choice, which will be used by the extension. These interpreters will have “start_XXX” and/or “end_XXX” methods where XXX is the ast node type you are interested in. They can also have also a “finish” method that is called after all python files have been analyzed.

Sample of UA extension code

import cast.analysers.ua
from cast import Event
from cast.analysers import log

from interpreters import Interpreter1, Interpreter2

class CustomPythonDiag(cast.analysers.ua.Extension):

    @Event('com.castsoftware.python', 'add_quality_rules')
    def add_quality_rules(self, externalQualityRules):

        # Add the interpreters
        externalQualityRules.add_interpreter(Interpreter1, self.get_plugin())
        externalQualityRules.add_interpreter(Interpreter2, self.get_plugin())
        ...

Sample of a simple interpreter code

class Interpreter:

    def __init__(self):
        pass

    def start_MethodCall(self, ast):
        ...

    def end_MethodCall(self, ast):
        ...

    def start_XXX(self, ast):
        ...

    def end_XXX(self, ast):
        ...

    ...

A complete sample is available here: Starter kits.

Python ast node types

Here we present the main classes and methods that form the API to exploit the information contained in the abstract syntax tree (AST) of a python file.

Basic elements

The AST returned by the Python analyzer is composed of nodes (Node type) and tokens (Token type). Tokens are the most basic elements. All nodes do contain sub-items (other sub-nodes and tokens). In some specific contexts one might not distinguish between nodes and tokens as they are treaded similarly.

All nodes inherit direcly or indirectly from light_parser.Node. Some nodes might inherit from some generic intermediate classes (Statement, BlockStatement, Term) used internally for parsing purposes. The character of the node (expression vs statement) is loosely correlated to the the name of these clases for certain nodes. These intermediate classes do not contribute to the API.

class light_parser.Token(text=None, type=None)

A token with code position.

get_begin_line()
Returns:the ast node begin line
Return type:int
get_begin_column()
Returns:the ast node begin column
Return type:int
get_end_line()
Returns:the ast node end line
Return type:int
get_end_column()
Returns:the ast node end column
Return type:int
get_type()
Returns:the name of the token type
Return type:str
get_children()

Convenient method to comply with ‘light_parser.Node’ interface.

Returns:empty list
get_sub_nodes()

Convenient method to comply with ‘light_parser.Node’ interface.

Returns:empty list
__eq__(self, other):

Equality.

Can compare to text directly. (case insensitive).

class light_parser.Node

An AST node

get_begin_line()
Returns:the ast node begin line
Return type:int
get_begin_column()
Returns:the ast node begin column
Return type:int
get_end_line()
Returns:the ast node end line
Return type:int
get_end_column()
Returns:the ast node end column
Return type:int
get_type()
Returns:the name of the node
Return type:str
get_type()
get_children()

An iterator on tokens and children that skips whitespaces and comments

get_sub_nodes(_type=None)

Return sub AST nodes

if a _type is provided returns only sub nodes of that type.

print_tree(depth=0)

Print as a tree.

The following intermediate classes inherit from Node, but do not contribute to the API.

class light_parser.Statement
class light_parser.BlockStatement
class light_parser.Term

Base classes

The classes below conveniently factorize common code among certain type of nodes. These are not explicit nodes of the AST.

class python_parser.PythonStatement
get_container_block()

Access to block, if, for etc... that contains that statement

@return: None for top level statements

class python_parser.PythonBlockStatement
get_statements()
class python_parser.WithDocString
get_docstring(unstripped=False)

Return python docstring of element

get_decorators()

Access to decorators of object.

@rtype: list of Decorator

class python_parser.WithExpression
get_expression(strip_parentheses=True)
class python_parser.PythonSimpleStatement
end = Or(;,Token.LineFeed)
class python_parser.Class

Bases: python_parser.PythonStatement, python_parser.WithDocString

Base class for Python classes

is_class()
Return type:boolean
get_name()

Access to class name

Return type:str
get_inheritance()

Access to inheritance

class python_parser.While
class python_parser.With

Common class for with statement

Statements

Definition statements

class python_parser.ClassBlock

Bases: python_parser.Class, light_parser.BlockStatement

Class definition containing an indented statement block

class python_parser.ClassOneLine

Bases: python_parser.Class, light_parser.Statement

Class definition in one line

class python_parser.FunctionBlock

Bases: python_parser.Function, light_parser.BlockStatement

Function definition containing an indented statement block

class python_parser.FunctionOneLine

Bases: python_parser.Function, light_parser.Statement

Function definition in one line

class python_parser.Global
class python_parser.NonLocal

Simple statements

class python_parser.ExpressionStatement

Bases: light_parser.Statement, python_parser.PythonSimpleStatement

get_expression()
is_expression_statement()
class python_parser.Import

Bases: light_parser.Statement, python_parser.PythonSimpleStatement

is_import()
get_module_references()

Access to the imported module

Returns:‘a.b’ for import a.b
get_imported_elements()

from ... import <imported elements>

class python_parser.Return

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Yield

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.YieldFrom

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Await

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Delete

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Break

Bases: light_parser.Statement, python_parser.PythonSimpleStatement

class python_parser.Continue

Bases: light_parser.Statement, python_parser.PythonSimpleStatement

class python_parser.Raise

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Print

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Exec

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Assert

Bases: light_parser.Statement, python_parser.PythonSimpleStatement, python_parser.WithExpression

class python_parser.Pass

Bases: light_parser.Statement, python_parser.PythonSimpleStatement

class python_parser.EllipsisObject

Bases: light_parser.Statement, python_parser.PythonSimpleStatement

Block statements

Statements containing other statements.

class python_parser.ForBlock

Bases: python_parser.PythonBlockStatement, python_parser.For

class python_parser.ForOneLine

Bases: light_parser.Statement, python_parser.For

class python_parser.WhileBlock

Bases: python_parser.PythonBlockStatement, python_parser.While

class python_parser.WhileOneLine

Bases: light_parser.Statement, python_parser.While

class python_parser.TryBlock

Bases: python_parser.PythonBlockStatement, python_parser.PythonStatement

class python_parser.TryOneLine

Bases: light_parser.Statement, python_parser.PythonStatement

class python_parser.ExceptBlock

Bases: python_parser.PythonBlockStatement, python_parser.PythonStatement, python_parser.WithExpression

class python_parser.ExceptOneLine

Bases: light_parser.Statement, python_parser.PythonStatement

class python_parser.WithBlock

Bases: python_parser.PythonBlockStatement, python_parser.With

A ‘with’ statement of the form:

with ... as ...:
stmt stmt ...
class python_parser.WithOneLine

Bases: light_parser.Statement, python_parser.With

with statement
with ... as ...: ...; ...;
get_statements()
class python_parser.FinallyBlock

Bases: python_parser.PythonBlockStatement, python_parser.PythonStatement

class python_parser.FinallyOneLine

Bases: light_parser.Statement, python_parser.PythonStatement

class python_parser.IfThenElseBlock

Bases: python_parser.PythonBlockStatement, python_parser.IfThenElse

Classical block

class python_parser.IfThenElseOneLine

Bases: light_parser.Statement, python_parser.IfThenElse

an if can be on one line : if .... : statement1; ... statementN;

get_statements()

Expressions

class python_parser.Identifier

Bases: python_parser.Expression

An identifier, e.g. : - a - f

No dots here, see DotAccess

get_name()
is_identifier()
get_assigned_expression()
get_resolution()

One might use this instead of ‘get_resolutions’ when it is highly unlikely to have ambiguity in resolution (or not important).

class python_parser.BinaryOperation

Bases: python_parser.Expression

A binary operation of the form:

left_expression op right_expression
get_left_expression()
get_right_expression()
is_binary_operation()
Return type:Boolean
get_operator()
is_assignment()
class python_parser.Assignment

Bases: python_parser.BinaryOperation

A assignment (includes augmented assignments)

augassign = ['=', '+=', '-=', '*=', '/=', '@=', '%=', '&=', '|=', '^=', '<<=', '>>=', '**=', '//=']
get_operator()
Return type:light_parser.Token
is_assignment()
get_assigned_variables()

Used for handling assignment destructuring

Limitation:
Unwrapping of parentheses not handled
get_assignee()

It returns the leftmost element in ‘get_assigned_variables’.

Use this for simple variable assignments of type
a = expression
Returns:Leftmost variable of the assignment
Return type:python_parser.Node (Identifier, ...)
class python_parser.Decorators

Bases: light_parser.Statement

A suite of decorators. We use this for matching the block of decorators.

class python_parser.Unpacking

Bases: python_parser.Expression

Unpacking (unary) operation like in *args, and **kwargs.

The name of the operator can be ‘*’ or ‘**’.

get_name()
class python_parser.Constant

Bases: python_parser.Expression

get_value()
is_constant()
is_constant_string()
is_fstring()
get_string_value()
class python_parser.DotAccess

Bases: python_parser.Expression

exp . accname

get_expression()
Returns:exp
Return type:Expression
get_identifier()
Returns:accname
Return type:Identifier
get_name()
Return type:str
is_dot_access()
class python_parser.ArrayAccess

Bases: python_parser.Expression

exp [...]

get_array_expression()
get_indice()
class python_parser.MethodCall

Bases: python_parser.Expression

exp (...)

get_method()

Return the expression without the parenthesis

f() -> f a.b() -> a.b ...
get_parameters()
is_method_call()
get_argument(position, keyword)

Returns positional/keyword argument

If found a match with positional argument, the keyword argument is neglected.

Limitations
Optional keyword arguments with no defined position can be retrieved by keyword. However combinations with optional (non-keyword) arguments are not tested.
is_resolved_to(qname)
class python_parser.Array

Bases: python_parser.Expression

An array expression :

[..., ..., ...]

get_values()
class python_parser.ComprehensionLoop

Bases: python_parser.Expression

[ ... for ... in ... ...] ( ... for ... in ... ...)

get_expression()

return first expression

get_comprehension_for()

Get the for part

class python_parser.ComprehensionFor

Bases: light_parser.Node, python_parser.For

The sub part of a comprehension loop

class python_parser.Map

Bases: python_parser.Expression, python_parser.WithResolution

A map expression:

{...:..., ...:..., ...:...}

get_values()
is_dict_like()
class python_parser.Lambda

Bases: python_parser.Expression

class python_parser.LambdaParameters

Bases: python_parser.Expression

class python_parser.Unary

Bases: python_parser.Expression

Unary expression

class python_parser.UnaryNot

Bases: python_parser.Unary

not expression

class python_parser.AdditionExpression

Bases: python_parser.BinaryOperation

exp + exp

is_addition()
class python_parser.StringInterpolation

Bases: python_parser.BinaryOperation

Expressions of type exp % exp

Limitations:
It over-captures modulo-operator expressions
is_interpolation()
class python_parser.IfTernaryExpression

Bases: python_parser.Expression

get_first_value()
get_second_value()
class python_parser.ExpressionList

Bases: python_parser.Expression

is_expression_list()

Other structural nodes

class python_parser.Parenthesis
class python_parser.SquareBracketedBlock
class python_parser.BracketedBlock
class python_parser.IndentBlock

Symbols

Symbols closely represent the structure of metamodel objects, among other functionalities. They are also used for resolution purposes for certain nodes.

class symbols.Symbol(name, parent=None)

base class for symbols

get_parent_symbol()
get_root_symbol()
get_name()

Return name

get_qualified_name()

Qualified name

get_ast()

Access to AST of symbol.

print_tree(depth=0)

Print as a tree.

get_statements()

Get the statements of the symbol.

get_class(name, begin_line=None)

Return a class by name. From the classes inside that symbol.

get_function(name, begin_line=None)

Return a function by name From the functions inside that symbol.

get_begin_line()
get_begin_column()
get_end_line()
get_end_column()
get_violations(property_name)

Returns all violations for a given rule.

add_violation(property_name, ast, *additional_asts)

Add a violation for a quality rule.

Parameters:
  • property_name – fullname of the property
  • ast – location of the violation
set_property(property_name, value)

Used to set a generic property on object

mainly used for quality rules

Parameters:
  • property_name – fullname of the property
  • value – value of the property
is_class()
is_function()
class symbols.Module(path, _file=None, text=None)

Bases: symbols.Symbol

A Python file

metamodel_type = 'CAST_Python_SourceCode'
is_package()

True when module is a package.

get_path()
get_file()
get_qualified_name()

Qualified name

get_statements()

Get the statements of the symbol.

get_main()

Return the ‘main’ found in that module

get_fullname()
find_local_symbols(name, types=[])

Override for modules.

class symbols.Class(name, parent)

Bases: symbols.Symbol

A python class

metamodel_type = 'CAST_Python_Class'
get_inheritance()

Inherited classes.

Return type:list of python_parser.Reference
get_ordered_inheritances()

Return the list of all inherited classes ordered by the Python Method Resolution Order (known as MRO, introduced in Python 2.3). Only those resolved classes are returned.

Return type:list of symbols.Class
find_method(name)

Find a method of the class, using inheritance

find_member(name)
is_callable(method)

Says that :param method: is callable on that class.

Handle override.

is_callable_by_name(name)

Says that :param method: is callable on that class.

Handle override.

get_fullname()
get_members()
get_classes()
get_functions()
is_class()
class symbols.Function(name, parent, return_type=None)

Bases: symbols.Symbol

A Python function or method

metamodel_type = 'CAST_Python_Method'
get_fullname()
is_function()

Writing tests

Sample of a test

import unittest
import cast.analysers.test
import cast.analysers.ua

class Test(unittest.TestCase):

    def test_ok1(self):

        analysis = cast.analysers.test.UATestAnalysis('PYTHON')  # set Python language
        analysis.add_selection('test1')
        analysis.add_dependency(r'C:\ProgramData\CAST\CAST\Extensions\com.castsoftware.python.1.4.0-beta2')
        analysis.run()

        f = analysis.get_object_by_name('f', 'CAST_Python_Function')
        self.assertTrue(f)
        value = f.get_value('CAST_Python_Rule.AvoidUsingInvalidObjectsInAll')
        self.assertEqual('1', value)


if __name__ == '__main__':
    unittest.main()

Note

You need to add a dependency by absolute path or relative to the folder containing com.castsoftware.python, for example

analysis.add_dependency(r'C:\ProgramData\CAST\CAST\Extensions\com.castsoftware.python.1.4.0-beta2')

For that you need to have downloaded the extension through CAST ExtensionDowloader: see https://doc.castsoftware.com/display/EXTEND/CAST+Extension+Downloader

Categories and types for quality rules

For adding a quality rule use the following categories

  • Category CAST_Python_Rule (for artifacts), CAST_Python_ClassRule (only for classes).

Quality rule scopes

Available basic quality rule scopes are :

1021000 Callable Python artifacts (includes functions methods and modules)
1021008 Python Functions (includes functions and methods)
1021011 Python Classes
1021012 Python Modules

Python technologies

Values for technology to be used in IMPLTechnologies.xml for filter value

Python 1021000