Running code after analyzers¶

Note

This is documentation for cast.application version 1.6.13. This API can be upgraded separately from AIP. See api 1.6.13

After all analysis job have ran some code can be launched on the resulting knowledge base.

During this step, you :

can create new links
can decorate existing objects or links with new properties
can create new violations on objects
has access to all objects and links created by any analyzer
have access to source code

See Choosing between analysis level and application level for choosing between the two levels

Warning

This API is an ORM.

So most of calls will perform a SQL query on the knowledge base. Thus classical bad practices still occurs here :

queries inside loops will destroy performances on large volume
...

See Special note on performances

Implementing An application level extension¶

A plugin can be called at the end of analysis by implementing a subclass of cast.application.ApplicationLevelExtension and overriding cast.application.ApplicationLevelExtension.end_application()

For example:

import cast.application
import logging

class ExtensionApplication(cast.application.ApplicationLevelExtension):

    def end_application(self, application):

        logging.debug("running code at the end of an application")

        # use application object to access the knowledge base
        # ...

It is better to separate code from application level extension from the analyser level extension. They use separate python API and separate process launching.

A plugin can contain any number of ApplicationLevelExtension, they all will be called back.

class cast.application.ApplicationLevelExtension¶

Inherit from this class to be consider as an application level extension.

Usage :

class MyExtension(ApplicationLevelExtension):

def end_application(self, application):

...

end_application_create_objects(application)¶

Called at the end of application’s analysis, the first pass to create new objects.

New in version 1.6.0.

end_application(application)¶

Called at the end of application’s analysis, the second pass to create new links.

Those call backs have the application as context cast.application.Application

Objects¶

Accessing objects¶

Examples:

# iterate on all objects of the application
for o in application.objects():
    logging.debug(o.get_name())
    # do something with the object

# iterate on all java objects of the application
for o in application.objects().has_type('Java'):
    logging.debug(o.get_name())
    # do something with the object

# iterate on all public java objects of the application
for o in application.objects().has_type('Java').is_public():
    logging.debug(o.get_name())
    # do something with the object
    ...

# iterate on all public java class of the application and load their body comment
for o in application.objects().has_type('Java').is_internal().is_class().is_public().load_property('comment.sourceCodeComment'):
    # do something with the object
    # the property is available because we have loaded it
    logging.debug(o.get_property('comment.sourceCodeComment'))

Objects can be accessed using cast.application.Application.objects(). This will be the preferred way for querying objects. It returns a cast.application.ObjectQuery that represent a query on objects. This query can be refined using method chaining.

Files, with their path loaded can be retrieved using cast.application.Application.get_files().

Objects from the applications will all be of one of the classes :

Object classification¶

We provide a high level classification of Objects :

Special note on performances¶

Here is a non complete list of bad practices found in code :

Query inside loop¶

Sample of bad practice:

for something in application.objects()...:

        for otherthing in application.objects()...:

                # ...

This will do a query inside a loop which is killer in term of performance.

Note that the following variation to the code above is similarly a bad practice:

objects1 = application.objects()...
objects2 = application.objects()...

for something in objects1:

    for otherthing in objects2:

       # ...

You probably should rewrite in the following form:

my_collection = {}

for something in application.objects()...:

        my_collection[...] = something

for otherthing in application.objects()...:

        # ... use my_collection in a correct way...

Do not use application.search_objects(name=variable)¶

Getting one object per name, whose name is variable will generate one query for each search and will be under-perfomant.

Creating new objects¶

For technologies where we do not have analysis level plugins available, you can create objects at application level with some restrictions.

everything should be done in the event cast.application.ApplicationLevelExtension.end_application_create_object() (creating, saving, saving property, saving links)
parent of a custom object is either an existing object or a saved custom object
saving property on them require object to be saved but does not require to use cast.application.Application.declare_property_ownership()
creating a link from or to those object require it to be saved

Example

custom = CustomObject()
custom.set_name('toto')
custom.set_parent(_file)
custom.set_type('C_CLASS')

custom.save()

custom.save_property('MyCategory.myProperty', 2)

logging.info('creating link...')
create_link('callLink', _file, custom)

class cast.application.CustomObject¶

For creating new objects at application level.

Bookmarks (position) cannot be saved for now.

set_name(name)¶

Set the object name.

Parameters:	name – object’s name. String

set_fullname(fullname)¶

Set the object’s fullname. Optional, if not provided, save() will calculate one : <parent’s full name>.<name>

Parameters:	name – object’s fullname. String

set_type(typename)¶

Set the type name.

Parameters:	typename – Metamodel type name. String

Type must respect some constraints for the object to be saved. ...

set_parent(parent)¶

Set the parent.

Parameters:	parent – the parent of type cast.application.Object

set_guid(guid)¶

Set the GUID of the object. Optional, if not provided, save() will calculate one : <typeid>?<fullname>

Generally speaking one does not have to bother about guids.

Parameters:	guid – object’s GUID. String

set_external()¶: Set object as external.

New in version 1.6.15.

save()¶

Save the object to Analysis Service.

Object require a non empty name, a saved parent and a type. The type must also follow constraints

save_property(prop, value)¶

Save a property on an object.

Parameters:	prop – the property to save. Either a string for the fullname of the property or an integer for the property id. value – property value. String or Integer

Object require to be saved.

Decorating objects¶

Exsiting objects can receive new:

properties by using cast.application.Object.save_property()
violations for quality rules by using cast.application.Object.save_violation()

In either case there are a certain number of preliminary steps to do :

having a property applicable for the metamodel type of the object either by :

reusing an existing property of the metamodel

declare a new property in the metamodel of the plugin

declare the handling of the property by calling cast.application.Application.declare_property_ownership()

Here is a complete sample code, assuming that the property exists:

# here say that we take in charge to put the value of the property 'CAST_SQL_Object.version' for
# all objects of type 'CAST_MSTSQL_RelationalTable'

application.declare_property_ownership('CAST_SQL_Object.version',['CAST_MSTSQL_RelationalTable'])

for o in application.objects().has_type('CAST_MSTSQL_RelationalTable'):
    o.save_property('CAST_SQL_Object.version', "my version")

Bookmark¶

All code positions are represented by objects of class cast.application.Bookmark

Through that class one can access file, file path lines and columns.

links¶

Accessing existing links¶

Already existing links (those created by analysers) can be accessed with cast.application.Application.links().

Which creates a cast.application.LinkQuery retrieving all links. This query can then be refined using the method chaining pattern API.

Individual link are of the type cast.application.EnlightenLink

Examples:

# iterate all links application
for link in application.links():
    ...

# iterate all call links of application
for link in application.links().has_type(LinkType.call):
    ...

# iterate all call links to java objects of application
for link in application.links().has_type(LinkType.call).has_callee(application.objects().has_type('Java')):
    ...

Decorating existing links with properties¶

The same way object can be decorated with new properties, existing links can receive new properties by using cast.application.EnlightenLink.save_property().

The property must have been declared to be handled by using cast.application.Application.declare_property_ownership()

Example:

links = application.links().has_type(LinkType.call).has_caller(application.objects().has_type('C/C++').is_virtual())

application.declare_property_ownership('physicalLink.inferenceEngineRequests', links)

for link in links:

    link.save_property('physicalLink.inferenceEngineRequests', ['virtual call'])
    logging.info('adding property on %s --> %s', link.get_caller().get_qualified_name(), link.get_callee().get_qualified_name())

Validating dynamic links¶

Not sure (also called dynamic links) can be marked as validated or ignored using:

Those methods will flag a link with the corresponding state provided they have been retrieved using cast.application.LinkQuery.is_not_sure().

Here is an example conversion from DLM Rule Engine to this API:

# all so called 'dynamic' links
for link in application.links().is_not_sure():

    callee = link.get_callee()

    expression = re.compile(...)

    if re.match(expresion,callee.get_fullname()):

        link.validate()

Creating links¶

New links can be created between two cast.application.Object by using cast.application.create_link()

Created links can receive properties :

save_property(prop, value)¶

Save a property on current link.

Parameters:	prop – the property to save. Either a string for the fullname of the property or an integer for the property id. value – the value to set, either a integer, a string or a list of those, it should respect the type declared in metamodel

Note

No need to declare property ownership for saving property on newly created links.

Example:

link = create_link('callLink', o1, o2)
link.save_property('physicalLink.inferenceEngineRequests', ['virtual call'])

Created links can receive any number of bookmarks :

add_bookmark(bookmark)¶

Parameters:	bookmark – a position for the link of type `cast.application.Bookmark`

Example:

link = create_link('callLink', o1, o2)
link.add_bookmark(bookmark1)
link.add_bookmark(bookmark2)
link.add_bookmark(bookmark3)

Created links can be marked as uncertain. They are then eligible to Dynamic Link Manager.

mark_as_not_sure()¶

Example:

link = create_link('callLink', o1, o2)
link.mark_as_not_sure() # indicate that this link is not certain

A reference finder¶

This API includes a simplified reference finder, cast.application.ReferenceFinder.

Warning

This API is different from the CAST-MS Reference Finder.

Here this API only ‘finds’ references. It creates no object, no links.

Typically : CAST-MS Reference Finder <=> cast.application.Application.get_files() + cast.application.ReferenceFinder + cast.application.Application.search_objects() + cast.application.create_link()

Basically an object of this class will search one or several patterns in a file, and returns an iterable of cast.application.Reference objects that represent the matches found.

Usage sample

rf = ReferenceFinder()

# declare a pattern named 'begin_pattern'
rf.add_pattern("begin_pattern", before="", element="begin", after="")
# one can add other patterns in the same reference finder
rf.add_pattern("end_pattern", before="", element="end", after="")

# we assume here that f is of type cast.application.File
references = list(rf.find_references_in_file(f))

This works as follow :

the file is opened
the text is scanned till one of the pattern matches (the first in the declaring order)
apply step 2 on the rest of the text ...

The result is represented by the following class. Please not that it creates no object in the knowledge base nor any object. The ‘reaction’ to those patterns found is totally open to developer.

class cast.application.Reference¶

A reference found in the code.

pattern_name Name of the found pattern

object object containing the reference

value the text that was matched

bookmark cast.application.Bookmark of the reference found

Reading files¶

This function will open a file while autodetecting encoding.

cast.application.open_source_file(path, encoding=None)¶

Equivalent of python open(path) that autotdetects encoding.

handles long path, UNC paths

Parameters:	encoding – specified encoding (optional)
Return type:	file

Logging¶

Use the standard python logging module:

import logging

class ExtensionApplication(cast.application.ApplicationLevelExtension):

    def end_application(self, application):

        logging.info("hello %s",  'world!')

Publishing reports¶

An extension can publish one or several reports at each global step. Those reports are then available both in CAST-MS and Console.

cast.application.publish_report(name, status, label, value, secondary_label=None, secondary_value=None, detail_report_path=None)¶

Publish a custom report that will be visible in CAST-MS.

@param name: str @param status: one of ‘OK’, ‘KO’, ‘Warning’ or None @param label: str @param value: str @param secondary_label: str or None @param secondary_value: str or None @param detail_report_path: str or None

Available in 8.3 and above.

New in version 1.5.12.

By using the status you can write custom sanity checks. By creating a result file you can also provide details about checks or actions.

Testing¶

Unit testing¶

Note

New in 1.5.13

Unit tests can be written the following way.

we create a fake knowledge base
we fill it with some data (objects, links etc...)
then we run our code to be tested
and finally we assert something on the result

For example, let’s say that you have a main.py containing a ApplicationLevelExtension and we want to test its end_application method:

import cast_upgrade_1_5_13
import unittest
import logging
from cast.application.test import TestKnowledgeBase # @UnresolvedImport
import main


class Test(unittest.TestCase):

    def test_run_extension(self):

        # 1. setup the initial state of the knowledge base
        test = TestKnowledgeBase()

        project = test.add_project('p1', 'JV_PROJECT')
        project.add_object('o1', 'o1', 'JV_CLASS')

        # 2. create an object of the app level extension
        extension = main.ExtensionApplication()

        # 3. run one method on the knowledge base
        application = test.run(extension.end_application, logging.DEBUG)

        # 4. assert something is done on the application
        ...

Setting up the initial state¶

The main entry point is to create a cast.application.test.TestKnowledgeBase instance

class cast.application.test.TestKnowledgeBase¶

A fake knowledge base meant for testing.

An application is already created inside.

Then to create one or more projects, which are result container for a given technology.

TestKnowledgeBase.add_project(name, _type)¶

Add a result project to the application. This is the result of an analysis. ‘Almost’ an analysis unit result.

Parameters: _type – int or str the metamodel type of the project

Then CAST objects can be created using :

TestObject.add_object(name, fullname, _type, internal=True, keyprop=0)¶

Create a child object.

Parameters:	_type – metamodel type int or str
Return type:	`TestObject`

TestObject.add_file(name, fullname, _type, content='', internal=True, filename=None)¶

Create child file object with content.

Parameters:	content – str the content of the file filename – str if given this will be the filename generated; or it can be an existing file.

A temporary file is generated with given content.

Links can be created using :

TestProject.add_link(_type, caller, callee)¶

Create a new link.

Parameters:	_type – cast.application.LinkType
Return type:	`TestLink`

Properties can be added on object and link using :

TestObject.add_property(prop_name_or_id, value)¶

Add a property on an object.

prop_name_or_id is the id of the metamodel property or its fullname, e.g., ‘comment.commentBeforeObject’ property must have INF_TYPE/INF_SUB_TYPE

Parameters:	prop_name_or_id – int or string

TestLink.add_property(prop_name_or_id, value)¶

Add a property on a link.

prop_name_or_id is the id of the metamodel property or its fullname, e.g., ‘comment.commentBeforeObject’ property must have INF_TYPE/INF_SUB_TYPE

Parameters:	prop_name_or_id – int or string

Running the code to be tested¶

Use the following method:

TestKnowledgeBase.run(method, log_level=None)¶: Run a function, or method having the application as parameter.

It accepts either :

a simple function with an application as parameter
or an instance method with an application as parameter

Example:

def simple_method(application):

“”” :param application: cast.application.Application “”“

Asserting the results¶

cast.application.test.TestKnowledgeBase.run() returns an object of type cast.application.Application so classical queries can be done to retreive results.

For example, a basic test of web linker

def test_01(self):

    test = TestKnowledgeBase()

    # a project
    project = test.add_project('toto', 'JV_PROJECT')

    # server part
    web_service = project.add_object('path2/path3/', 'path2/path3/', 'CAST_JAXRS_GetOperation')

    # client part
    call_to_webservice = project.add_object('caller', 'call', 'CAST_JQuery_GetResourceService')
    call_to_webservice.add_property('CAST_ResourceService.uri', '/path1/path2/path3/')

    extension = ExtensionApplication()

    application = test.run(extension.end_application)

    caller = list(application.search_objects(name='caller'))[0]
    callee = list(application.search_objects(name='path2/path3/'))[0]

    links = application.links().has_caller([caller]).has_callee([callee])

    self.assertEqual(1, len(list(links)))

What cannot be tested this way¶

The following elements cannot be tested this way :

Running on an existing schema¶

You can run your application level code on an existing schema using the following function.

test.run(kb_name, application_name, engine=None, event='end_application')¶

Run current app level plugin on a kb/application.

The code calling this function should be located in a tests subfolder of the plugin.

Parameters:	engine – sqlalchemy engine, see `cast.application.create_postgres_engine()`

For example

run(kb_name='illustration_local', application_name='Illustration')

Using legacy tools¶

For the sake of continuity we have kept some of the tools after analysis one can find in CAST-MS.

Warning

Those are kept for compatibility. But it is strongly recommended to use programmatic API when possible.

Running ‘tools after analysis’¶

The so called ‘Cast-MS tools after analysis’ are generally used to create missing links from frameworks. They can be directly used from inside the plugin by using :

Note that it is strongly recommended to use ansi sql code here. The SQL code will be executed in the correct schema so following code is perfectly working:

class ExtensionApplication(cast.application.ApplicationLevelExtension):

    def end_application(self, application):

        application.update_cast_knowledge_base("""
        insert into CI_LINKS (CALLER_ID, CALLED_ID, LINK_TYPE, ERROR_ID) values (1,1,1,0);
        """)

So no need to prefix tables with nothing.

Access to infrastructure and management¶

Warning

This only works on ‘combined install’ situation; i.e., the local schema must be named xxx_local and the management xxx_mngt both on the same server.

cast.application.Application.get_application_configuration()

cast.application.Application.get_managment_base()

Management schema¶

Management schema contains the analysis definitions and options.

class cast.application.managment.ManagmentBase(name, engine=None)¶

A connection to a management base

get_applications()¶

Get the applications defined in the base.

Return type:	list of `cast.application.managment.Application`

get_application(name)¶

Access to an application by name.

Parameters:	name (str) – the name of the application
Return type:	`cast.application.managment.Application`

get_delivery_path()¶

Access to delivery path.

Return type:	str

get_deploy_path()¶

Access to deploy path.

Return type:	str

get_dmt_application(name)¶

Access to an application in DMT inside the delivery folder.

Return type:	cast.application.DMTApplication

New in version 1.6.17.

execute_query(query)¶: Execute a query on the schema and return a cursor.

New in version 1.5.4.

get_caip_version()¶

Access to CAIP version of the schema.

Returns:	distutils.version.StrictVersion

get_extensions()¶: Access to extensions installed on the schema.

New in version 1.5.4.

get_install_events()¶: Access to install events of the schema.

New in version 1.5.4.

get_size()¶: Get the schema size in bytes.

update_license_key(license_key)¶: Change the license key

New in version 1.6.21.

class cast.application.managment.Application¶

The definition of an application in CMS.

get_packages()¶

Get the source packages of the application. :rtype: list of cast.application.managment.Package

get_analysis_units()¶: Get the analysis units of the application. :rtype: list of cast.application.managment.AnalysisUnit

get_modules()¶: Access to modules definitions :rtype: list of cast.application.managment.Module

get_module(name)¶: Get a module definition :rtype: cast.application.managment.Module

get_snapshots()¶: Get the snapshots of the application. :rtype: list of cast.application.managment.Snapshot

get_analysis_service()¶: Access to analysis service of the application :rtype: cast.application.KnowledgeBase

get_dashboard_services()¶: Access to analysis service of the application :rtype: list of cast.application.central.CentralBase

get_email_to_send_reports()¶: If configured, returns the email to send reports to. :rtype: str

class cast.application.managment.Package(identifier, name, path, _type)¶

A source code package.

get_name()¶: Name of the package

get_path()¶: Deployment path of the package

get_type()¶

Package type.

Returns:	‘bo’, ‘dbudb’, ‘dbzosextract’, ‘file’, ‘formsextract’, ‘mfextract’, ‘oracleextract’, ‘sapextract’, ‘sqlserverextract’, ‘sybaseextract’, ‘uiextract’

class cast.application.managment.AnalysisUnit(mb, identifier, name, technology, application, log, last_execution_date, uuid)¶

The definition of an analysis unit.

get_technology()¶

Access to technology of the AU.

Returns:	‘asp’, ‘bo’, ‘cobol’, ‘cpp’, ‘forms’, ‘j2ee’, ‘net’, ‘ora’, ‘pb’, ‘sap’, ‘sqlsrv’, ‘syb’, ‘ua’, ‘udb’, ‘ui’, ‘vb’, ‘zos’

get_technologies()¶: Generalized access to the technologies present in that AU. For ‘core’ technology return a list of one element, e.g., : [‘cpp’] [‘j2ee’] For Universal Analyzer return a list of selected technologies (there may be several in one UA analysis unit).

New in version 1.5.20.

get_included_selection()¶: Return the pathes/folders selected as included in CMS.

New in version 1.5.20.

get_excluded_selection()¶: Return the pathes/folders selected as excluded in CMS.

New in version 1.5.20.

get_log()¶: Last analysis log path.

get_last_execution_date()¶: Last execution date.

projects(kb)¶: Given a KnowledgeBase, gives the associated projects

heads(kb)¶: Given a KnowledgeBase, gives the associated head.

full(kb)¶: Given a KnowledgeBase, gives the associated full content

jobs(kb)¶

Given a KnowledgeBase, gives the job associated with analysis unit.

Probably bugged...

class cast.application.managment.Module(mb, identifier, name, uuid, application)¶: The definition of a module

Interactive scripting¶

The API can also be used outside plugins, in a script or in an interactive manner.

Basically one has to ‘connect’ to a knowledge base and then can retrieve applications, etc..

class cast.application.KnowledgeBase(name, engine=None)¶

A connection to a knowledge base.

get_applications()¶

Returns the list of application of the knowledge base.

Return type:	list of `cast.application.Application`

get_application(name)¶

Access to an application by name.

Parameters:	name (str) – the name of the application
Return type:	`cast.application.Application`

get_jobs()¶: Access to jobs of the knowledge base.

get_events()¶

Get the events of this KnowledgeBase.

AMT Saving
Metrics Calculation (after the analysis)
Update Cast Knowledge Base
Installation, upgrade events
...

New in version 1.5.4.

update_cast_system_views()¶: Updates the Cast System Views.

execute_query(query)¶: Execute a query on the schema and return a cursor.

New in version 1.5.4.

get_caip_version()¶

Access to CAIP version of the schema.

Returns:	distutils.version.StrictVersion

get_extensions()¶: Access to extensions installed on the schema.

New in version 1.5.4.

get_install_events()¶: Access to install events of the schema.

New in version 1.5.4.

get_size()¶: Get the schema size in bytes.

update_license_key(license_key)¶: Change the license key

New in version 1.6.21.

Example of a script:

from cast.application import KnowledgeBase, LinkType

kb = KnowledgeBase('b800_8525_local')

print(kb.get_applications())

application = kb.get_application(name='Cpp')

for link in application.links().has_type([LinkType.call, LinkType.accessExec]).has_callee(application.objects().is_virtual()):

    print(link)

Default connection is a css localhost:2280. But any other connection can be passed to the constructor of cast.application.KnowledgeBase

cast.application.create_postgres_engine(user='operator', password='CastAIP', host='localhost', port=2280, database='postgres')¶

cast.application.create_oracle_engine(user, password, host, port, sid, service)¶: sid or service name

cast.application.create_sqlserver_engine(user, password, server, schema, port, instance, trusted=False)¶

Example:

from cast.application import KnowledgeBase, LinkType, create_oracle_engine

engine = create_oracle_engine(user="EXTKB734", password="EXTKB734", host="JNK2O11", port=1521, sid="JNK2O11", service=None)

kb = KnowledgeBase('extkb734', engine=engine)

...