Running code after analyzers ============================ .. note:: This is documentation for cast.application version 1.6.13. This API can be upgraded separately from AIP. See `api 1.6.13 `_ After all analysis job have ran some code can be launched on the resulting knowledge base. During this step, you : * can create new links * can decorate existing objects or links with new properties * can create new violations on objects * has access to all objects and links created by any analyzer * have access to source code See :ref:`choosing-level` for choosing between the two levels .. warning:: This API is an `ORM `_. So most of calls will perform a SQL query on the knowledge base. Thus **classical bad practices** still occurs here : * queries inside loops will destroy performances on large volume * ... See :ref:`performance-notes` Implementing An application level extension ------------------------------------------- A plugin can be called at the end of analysis by implementing a subclass of :class:`cast.application.ApplicationLevelExtension` and overriding :func:`cast.application.ApplicationLevelExtension.end_application` For example:: import cast.application import logging class ExtensionApplication(cast.application.ApplicationLevelExtension): def end_application(self, application): logging.debug("running code at the end of an application") # use application object to access the knowledge base # ... It is better to separate code from application level extension from the analyser level extension. They use separate python API and separate process launching. A plugin can contain any number of ApplicationLevelExtension, they all will be called back. .. autoclass:: cast.application.ApplicationLevelExtension .. automethod:: cast.application.ApplicationLevelExtension.end_application_create_objects .. automethod:: cast.application.ApplicationLevelExtension.end_application Those call backs have the application as context :class:`cast.application.Application` Objects ------- Accessing objects ***************** Examples:: # iterate on all objects of the application for o in application.objects(): logging.debug(o.get_name()) # do something with the object # iterate on all java objects of the application for o in application.objects().has_type('Java'): logging.debug(o.get_name()) # do something with the object # iterate on all public java objects of the application for o in application.objects().has_type('Java').is_public(): logging.debug(o.get_name()) # do something with the object ... # iterate on all public java class of the application and load their body comment for o in application.objects().has_type('Java').is_internal().is_class().is_public().load_property('comment.sourceCodeComment'): # do something with the object # the property is available because we have loaded it logging.debug(o.get_property('comment.sourceCodeComment')) Objects can be accessed using :func:`cast.application.Application.objects`. This will be the preferred way for querying objects. It returns a :class:`cast.application.ObjectQuery` that represent a query on objects. This query can be refined using `method chaining `_. Files, with their path loaded can be retrieved using :func:`cast.application.Application.get_files`. Objects from the applications will all be of one of the classes : * :class:`cast.application.Object` * :class:`cast.application.File` We provide a high level classification of Objects : * :func:`cast.application.Object.is_file` * :func:`cast.application.Object.is_program` * :func:`cast.application.Object.is_class` * :func:`cast.application.Object.is_executable` * :func:`cast.application.Object.is_package` * :func:`cast.application.Object.is_variable` * :func:`cast.application.Object.is_table` * :func:`cast.application.Object.is_index` * :func:`cast.application.Object.is_foreignkey` * :func:`cast.application.Object.is_form` * :func:`cast.application.Object.is_web_service_operation` * :func:`cast.application.Object.is_dbms` .. _performance-notes: Special note on performances **************************** Here is a non complete list of bad practices found in code : Query inside loop ################# Sample of bad practice:: for something in application.objects()...: for otherthing in application.objects()...: # ... This will do a query inside a loop which is killer in term of performance. You probably should rewrite in the following form:: my_collection = {} for something in application.objects()...: my_collection[...] = something for otherthing in application.objects()...: # ... use my_collection in a correct way... Do not use application.search_objects(name=variable) #################################################### Getting one object per name, whose name is variable will generate one query for each search and will be under-perfomant. Creating new objects ******************** For technologies where we do not have analysis level plugins available, you can create objects at application level with some restrictions. 1. everything should be done in the event :func:`cast.application.ApplicationLevelExtension.end_application_create_object` (creating, saving, saving property, saving links) 2. parent of a custom object is either an existing object or a saved custom object 3. saving property on them require object to be saved but **does not require to use** :func:`cast.application.Application.declare_property_ownership` 4. creating a link from or to those object require it to be saved Example :: custom = CustomObject() custom.set_name('toto') custom.set_parent(_file) custom.set_type('C_CLASS') custom.save() custom.save_property('MyCategory.myProperty', 2) logging.info('creating link...') create_link('callLink', _file, custom) .. autoclass:: cast.application.CustomObject :members: Decorating objects ****************** Exsiting objects can receive new: * properties by using :func:`cast.application.Object.save_property` * violations for quality rules by using :func:`cast.application.Object.save_violation` In either case there are a certain number of preliminary steps to do : 1. having a property applicable for the metamodel type of the object either by : a. reusing an existing property of the metamodel b. declare a new property in the metamodel of the plugin 2. declare the handling of the property by calling :func:`cast.application.Application.declare_property_ownership` Here is a complete sample code, assuming that the property exists:: # here say that we take in charge to put the value of the property 'CAST_SQL_Object.version' for # all objects of type 'CAST_MSTSQL_RelationalTable' application.declare_property_ownership('CAST_SQL_Object.version',['CAST_MSTSQL_RelationalTable']) for o in application.objects().has_type('CAST_MSTSQL_RelationalTable'): o.save_property('CAST_SQL_Object.version', "my version") Bookmark ******** All code positions are represented by objects of class :class:`cast.application.Bookmark` Through that class one can access file, file path lines and columns. links ----- Accessing existing links ************************ Already existing links (those created by analysers) can be accessed with :func:`cast.application.Application.links`. Which creates a :class:`cast.application.LinkQuery` retrieving all links. This query can then be refined using the method chaining pattern API. Individual link are of the type :class:`cast.application.EnlightenLink` Examples:: # iterate all links application for link in application.links(): ... # iterate all call links of application for link in application.links().has_type(LinkType.call): ... # iterate all call links to java objects of application for link in application.links().has_type(LinkType.call).has_callee(application.objects().has_type('Java')): ... Decorating existing links with properties ***************************************** The same way object can be decorated with new properties, existing links can receive new properties by using :func:`cast.application.EnlightenLink.save_property`. The property must have been declared to be handled by using :meth:`cast.application.Application.declare_property_ownership` Example:: links = application.links().has_type(LinkType.call).has_caller(application.objects().has_type('C/C++').is_virtual()) application.declare_property_ownership('physicalLink.inferenceEngineRequests', links) for link in links: link.save_property('physicalLink.inferenceEngineRequests', ['virtual call']) logging.info('adding property on %s --> %s', link.get_caller().get_qualified_name(), link.get_callee().get_qualified_name()) Validating dynamic links ************************ Not sure (also called dynamic links) can be marked as validated or ignored using: * :func:`cast.application.EnlightenLink.validate` * :func:`cast.application.EnlightenLink.ignore` Those methods will flag a link with the corresponding state provided they have been retrieved using :func:`cast.application.LinkQuery.is_not_sure`. Here is an example conversion from DLM Rule Engine to this API:: # all so called 'dynamic' links for link in application.links().is_not_sure(): callee = link.get_callee() expression = re.compile(...) if re.match(expresion,callee.get_fullname()): link.validate() Creating links ************** New links can be created between two :class:`cast.application.Object` by using :func:`cast.application.create_link` Created links can receive properties : .. method:: save_property(prop, value) Save a property on current link. :param prop: the property to save. Either a string for the fullname of the property or an integer for the property id. :param value: the value to set, either a integer, a string or a list of those, it should respect the type declared in metamodel .. note:: No need to declare property ownership for saving property on newly created links. Example:: link = create_link('callLink', o1, o2) link.save_property('physicalLink.inferenceEngineRequests', ['virtual call']) Created links can receive any number of bookmarks : .. method:: add_bookmark(bookmark) :param bookmark: a position for the link of type :class:`cast.application.Bookmark` Example:: link = create_link('callLink', o1, o2) link.add_bookmark(bookmark1) link.add_bookmark(bookmark2) link.add_bookmark(bookmark3) Created links can be marked as uncertain. They are then eligible to Dynamic Link Manager. .. method:: mark_as_not_sure() Example:: link = create_link('callLink', o1, o2) link.mark_as_not_sure() # indicate that this link is not certain A reference finder ------------------ This API includes a simplified reference finder, :class:`cast.application.ReferenceFinder`. .. warning:: This API is different from the CAST-MS Reference Finder. Here this API only 'finds' references. It creates no object, no links. Typically : CAST-MS Reference Finder <=> :meth:`cast.application.Application.get_files` + :class:`cast.application.ReferenceFinder` + :meth:`cast.application.Application.search_objects` + :meth:`cast.application.create_link` Basically an object of this class will search one or several patterns in a file, and returns an iterable of :class:`cast.application.Reference` objects that represent the matches found. Usage sample :: rf = ReferenceFinder() # declare a pattern named 'begin_pattern' rf.add_pattern("begin_pattern", before="", element="begin", after="") # one can add other patterns in the same reference finder rf.add_pattern("end_pattern", before="", element="end", after="") # we assume here that f is of type cast.application.File references = list(rf.find_references_in_file(f)) This works as follow : 1. the file is opened 2. the text is scanned till one of the pattern matches (the first in the declaring order) 3. apply step 2 on the rest of the text ... The result is represented by the following class. Please not that it creates no object in the knowledge base nor any object. The 'reaction' to those patterns found is totally open to developer. .. class:: cast.application.Reference A reference found in the code. `pattern_name` Name of the found pattern `object` object containing the reference `value` the text that was matched `bookmark` :class:`cast.application.Bookmark` of the reference found Reading files ------------- This function will open a file while autodetecting encoding. .. autofunction:: cast.application.open_source_file Logging ------- Use the standard python `logging `_ module:: import logging class ExtensionApplication(cast.application.ApplicationLevelExtension): def end_application(self, application): logging.info("hello %s", 'world!') Publishing reports ------------------ An extension can publish one or several reports at each global step. Those reports are then available both in CAST-MS and Console. .. autofunction:: cast.application.publish_report By using the status you can write custom sanity checks. By creating a result file you can also provide details about checks or actions. Testing ------- Unit testing ************ .. note:: New in 1.5.13 Unit tests can be written the following way. * we create a fake knowledge base * we fill it with some data (objects, links etc...) * then we run our code to be tested * and finally we assert something on the result For example, let's say that you have a main.py containing a ApplicationLevelExtension and we want to test its end_application method:: import cast_upgrade_1_5_13 import unittest import logging from cast.application.test import TestKnowledgeBase # @UnresolvedImport import main class Test(unittest.TestCase): def test_run_extension(self): # 1. setup the initial state of the knowledge base test = TestKnowledgeBase() project = test.add_project('p1', 'JV_PROJECT') project.add_object('o1', 'o1', 'JV_CLASS') # 2. create an object of the app level extension extension = main.ExtensionApplication() # 3. run one method on the knowledge base application = test.run(extension.end_application, logging.DEBUG) # 4. assert something is done on the application ... Setting up the initial state ############################ The main entry point is to create a :class:`cast.application.test.TestKnowledgeBase` instance .. autoclass:: cast.application.test.TestKnowledgeBase Then to create one or more projects, which are result container for a given technology. .. automethod:: cast.application.test.TestKnowledgeBase.add_project Then CAST objects can be created using : .. automethod:: cast.application.test.TestObject.add_object .. automethod:: cast.application.test.TestObject.add_file Links can be created using : .. automethod:: cast.application.test.TestProject.add_link Properties can be added on object and link using : .. automethod:: cast.application.test.TestObject.add_property .. automethod:: cast.application.test.TestLink.add_property Running the code to be tested ############################# Use the following method: .. automethod:: cast.application.test.TestKnowledgeBase.run It accepts either : * a simple function with an application as parameter * or an instance method with an application as parameter Example: def simple_method(application): """ :param application: :class:`cast.application.Application` """ Asserting the results ##################### :meth:`cast.application.test.TestKnowledgeBase.run` returns an object of type :class:`cast.application.Application` so classical queries can be done to retreive results. For example, a basic test of web linker :: def test_01(self): test = TestKnowledgeBase() # a project project = test.add_project('toto', 'JV_PROJECT') # server part web_service = project.add_object('path2/path3/', 'path2/path3/', 'CAST_JAXRS_GetOperation') # client part call_to_webservice = project.add_object('caller', 'call', 'CAST_JQuery_GetResourceService') call_to_webservice.add_property('CAST_ResourceService.uri', '/path1/path2/path3/') extension = ExtensionApplication() application = test.run(extension.end_application) caller = list(application.search_objects(name='caller'))[0] callee = list(application.search_objects(name='path2/path3/'))[0] links = application.links().has_caller([caller]).has_callee([callee]) self.assertEqual(1, len(list(links))) What cannot be tested this way ############################## The following elements cannot be tested this way : * :func:`cast.application.Application.sql_tool` * :func:`cast.application.Application.update_cast_knowledge_base` Running on an existing schema ***************************** You can run your application level code on an existing schema using the following function. .. automethod:: cast.application.test.run For example :: run(kb_name='illustration_local', application_name='Illustration') Using legacy tools ------------------ For the sake of continuity we have kept some of the tools after analysis one can find in CAST-MS. .. warning:: Those are kept for compatibility. But it is strongly recommended to use programmatic API when possible. Running 'tools after analysis' ****************************** The so called 'Cast-MS tools after analysis' are generally used to create missing links from frameworks. They can be directly used from inside the plugin by using : * :func:`cast.application.Application.sql_tool` * :func:`cast.application.Application.update_cast_knowledge_base` Note that it is strongly recommended to use ansi sql code here. The SQL code will be executed in the correct schema so following code is perfectly working:: class ExtensionApplication(cast.application.ApplicationLevelExtension): def end_application(self, application): application.update_cast_knowledge_base(""" insert into CI_LINKS (CALLER_ID, CALLED_ID, LINK_TYPE, ERROR_ID) values (1,1,1,0); """) So no need to prefix tables with nothing. Access to infrastructure and management --------------------------------------- .. warning:: This only works on 'combined install' situation; i.e., the local schema must be named xxx_local and the management xxx_mngt both on the same server. :func:`cast.application.Application.get_application_configuration` :func:`cast.application.Application.get_managment_base` Management schema ----------------- Management schema contains the analysis definitions and options. .. autoclass:: cast.application.managment.ManagmentBase :members: :inherited-members: .. class:: cast.application.managment.Application The definition of an application in CMS. .. method:: get_packages() Get the source packages of the application. :rtype: list of :class:`cast.application.managment.Package` .. method:: get_analysis_units() Get the analysis units of the application. :rtype: list of :class:`cast.application.managment.AnalysisUnit` .. method:: get_modules() Access to modules definitions :rtype: list of :class:`cast.application.managment.Module` .. method:: get_module(name) Get a module definition :rtype: :class:`cast.application.managment.Module` .. method:: get_snapshots() Get the snapshots of the application. :rtype: list of :class:`cast.application.managment.Snapshot` .. method:: get_analysis_service() Access to analysis service of the application :rtype: :class:`cast.application.KnowledgeBase` .. method:: get_dashboard_services() Access to analysis service of the application :rtype: list of :class:`cast.application.central.CentralBase` .. method:: get_email_to_send_reports() If configured, returns the email to send reports to. :rtype: str .. autoclass:: cast.application.managment.Package :members: .. autoclass:: cast.application.managment.AnalysisUnit :members: .. autoclass:: cast.application.managment.Module :members: Interactive scripting --------------------- The API can also be used outside plugins, in a script or in an interactive manner. Basically one has to 'connect' to a knowledge base and then can retrieve applications, etc.. .. autoclass:: cast.application.KnowledgeBase :members: :inherited-members: Example of a script:: from cast.application import KnowledgeBase, LinkType kb = KnowledgeBase('b800_8525_local') print(kb.get_applications()) application = kb.get_application(name='Cpp') for link in application.links().has_type([LinkType.call, LinkType.accessExec]).has_callee(application.objects().is_virtual()): print(link) Default connection is a css localhost:2280. But any other connection can be passed to the constructor of :class:`cast.application.KnowledgeBase` .. autofunction:: cast.application.create_postgres_engine .. autofunction:: cast.application.create_oracle_engine .. autofunction:: cast.application.create_sqlserver_engine Example:: from cast.application import KnowledgeBase, LinkType, create_oracle_engine engine = create_oracle_engine(user="EXTKB734", password="EXTKB734", host="JNK2O11", port=1521, sid="JNK2O11", service=None) kb = KnowledgeBase('extkb734', engine=engine) ...