Saturday, March 19, 2016

How to enable Grails application to send SMS via Twilio API

How to send SMS with Grails Application via Twilio



 

Are you trying to send text messages from your Grails web application? If your answer is Yes then you are in luck. If your answer is No .. well... stick around... you'll learn something. 

Target audience:

People interested in software development, Grails framework and Groovy programming language.
This article may be more interesting if you already know Grails framework. 

What you need:

  1. A twilio account with your API keys
  2. The grails-twilio plugin
  3. Your grails application ( Grails version 2.x and above )

To create a twilio account, please visit http://www.twilio.com. Copy your API Keys
The twilio-grails plug-in provides sms sending capability to a Grails application via twilio api.
Configuration

Add twilio plugin

Depending on what version of Grails you are using, open your buildConfig.groovy file or build.gradle file and add the following line.

compile "org.grails.plugins:twilio:0.1"
for grails 2.x

and 

compile "org.grails.plugins:twilio:0.1.0"
For grails 3.x

Add twilio properties

Add your twilio properties to grails configuration file: The config file for grails 2.x is Config.groovy. For Grails 3.x users, please add the following to your application.groovy file or application.yml ( Use yml syntax )
twilio {
    // Enter your host address
    host = 'https://api.twilio.com'
    apiID = 'enter your api Id'
    apiPass = 'enter your api password'
    smsUrl = '/2010-04-01/Accounts/' + apiID + '/Messages.json'
    number = ""
}

Add more dependencies

Copy and paste the following to your BuildConfig.groovy or Build.gradle file 
 compile(group:'org.apache.httpcomponents',name:'httpclient',version:'4.3.6') compile(group:'org.apache.httpcomponents',name:'fluent-hc',version:'4.3.6') compile(group:'org.apache.httpcomponents',name:'httpclient-cache',version:'4.3.6') compile(group:'org.apache.httpcomponents',name:'httpmime',version:'4.3.6') 

Create your controller

Create the grails controller you wish to handle the SMS sending flow


Inject smsService into your controller

Inject smsService into your class
def smsService
smsService is a Grails service provide by the twilio plugin for easy sending SMSs. 
It provides a method called send() that can take mapped parameters. Please note that 'send()' is overloaded 'see http://en.wikipedia.org/wiki/Function_overloading' and can take various variations of parameters.

One simple form is:  send(Map map) 
Where ......
map contains parameters;
map['phone'] :  The phone number of recipient eg +1234444444
map['from'] : Your twilio assigned number eg. +09899898989
map['body']:  "The body of your message" map.mediaUrl: "Url for any attachment" (optional )

Example Usage


An example usage can be seen below.
Class YourController{

    def smsService
    ...
    def yourMethod(){
        def map = [to:"070987878787",from:"09808000000",body:"SMS BODY"]
        smsService.send(map)

    }

}

You are done.

Thursday, March 3, 2016

How to accept PayPal payments on Grails

How to accept PayPal payments
with Grails Application.







Introduction to Paypal payments plugin.

The Paypal payments plugin simplifies the integration of Paypal

with Grails applications.  Grails is a web framework for developing web applications and a

plugin is a module that can be easily installed to add specific

features to your application. Modules/Plugins are usually designed

to accomplish a specific task. 

Who should read this?


Anyone interested in how to integrate paypal with a Grails application. 

What you need

  1. A Paypal developer account with your API keys
  2. Grails paypal payments plugin
  3. Grails Application ( Version 2.x and above ) 
If you do not already have a Paypal developer account please visit https://developer.paypal.com/developer/accounts/ to get one. 

The paypal-payments plugin will provide the features required to interact with the Paypal API. You do not need to be a paypal payments guru to use Paypal. 

A Grails application is required. A demo app is just fine. 

Create a Paypal sandbox account.

After you have created your Paypal developer account, Create a REST API app on your Paypal user dashboard. The app will enable you to receive REST API credentials for testing and live transactions.

The test environment (Sandbox) will allow you to play around with most of the API features available. It is also a good way to test your application integration before moving over to a live environment.

Create your Grails Application.

I will assume that you already have your Grails application designed and ready to accept payments. A demo application will be sufficient for this exercise. A discussion on how to  create a Grails application is beyond the scope of this post. I will address it in future posts ;)


Integrate Grails paypal-payments plugin


You need to add the paypal payments plugin to your grails dependencies build.gradle (Grails 3.x) or BuildConfig.groovy file ( Grails 2.x). Inside your dependencies block add the following: 

compile 'org.grails.plugins:paypal-payments:0.1.0'

Obtain your client id and client secret from Paypal dashboard


Obtain the API keys for your sandbox account/app and add it to your grails config file [Config.groovy for grails 2.x and application.groovy for grails 3.x].

Here's what my config file looks like:

paypal.email="myname@mydomain.com"
paypal.clientId = 'your client id'
paypal.sandbox.clientId = 'your client id'
paypal.clientSecret = 'your client secret'
paypal.sandbox.clientSecret ='your client secret'
paypal.endpoint = "https://api.paypal.com"
paypal.sandbox.endpoint = "https://api.sandbox.paypal.com"


Notice that I added config for sandbox and live environment. The reason is that I want to be able to switch between both environments during app development.

Create a Grails controller and add required actions

Create a Grails controller. Personally, I called my own controller PaypalController. There is no need to create another controller if you already have one you want to use. 

To create a Grails controller, go to your app folder and type:

grails create-controller com.mypackage.Paypal

Inject PaypalService

The paypal payments make available, a PaypalService. The PaypalService provides methods for accomplishing common tasks with Paypal API.

Inject the service into your controller. 

Class PaypalController{    
    def paypalService
   ....
   ...
  
}


Add required actions

I like to complete the Paypal payments collection in 3 steps.  I think it normally takes three steps :)

Here are the steps:

1. Make approval request to paypal.
2. Approval - Customer approves the payment
3. Execution - Process response from Paypal in order to execute the payment.

Approval Step


For the approval step, you need to redirect the user/customer to the paypal website in order to approve the transaction. The approval usually involves the user entering payment information that will allow Paypal to proceed with the transaction.

To complete the approval step here's my action inside my PaypalController .


def approve(){      

    String clientId = grailsApplication.config.paypal.clientId
    String clientSecret = grailsApplication.config.paypal.clientSecret
    String endpoint = grailsApplication.config.paypal.endpoint
    Map sdkConfig = [:] 
    sdkConfig.put(Constants.CLIENT_ID,clientId)
    sdkConfig.put(Constants.CLIENT_SECRET,clientSecret)
    sdkConfig.put(Constants.ENDPOINT,endpoint)
    def accessToken = paypalService.getAccessToken(clientId,clientSecret,sdkConfig)
    def apiContext = paypalService.getAPIContext(accessToken,sdkConfig)

    def details = paypalService.createDetails(['subtotal':"12.50"])
    def amount = paypalService.createAmount(['currency':currencyCode,'total':"12.50",'details':details])

    def transaction = paypalService.createTransaction(['amount':amount,'description':"your description",details:details])
    def transactions = []
    transactions.add(transaction)

    def payer = paypalService.createPayer(['paymentMethod':'paypal'])
    def cancelUrl="http://mydomain/paypal/cancel";
    def returnUrl = "http://mydomain/paypal/execute";

    def redirectUrls = paypalService.createRedirectUrls(['cancelUrl':cancelUrl,'returnUrl':returnUrl])


    def payment
    try{
        // create the paypal payment
        payment = paypalService.createPayment(['payer':payer,'intent':'sale'
                ,'transactionList':transactions,'redirectUrls':redirectUrls
                ,'apiContext':apiContext])



    }
    catch(Exception ex){
        String msg = ex.getMessage()
        flash.message = "Could not complete the transaction because: ${msg? msg : ''}"  

        redirect controller:'paypal', action:"error"
        return
    }

    def approvalUrl = ""
    def retUrl = ""
    // retrieve links from returned paypal object
    payment?.links.each{ 
        if(it?.rel == 'approval_url'){
            approvalUrl = it.href
        }
        if(it?.rel == 'return_url'){
            retUrl = it.href

        }
    }


    redirect url:approvalUrl? approvalUrl:'/', method:'POST'

}


Approval

The customer will be redirected to the Paypal website for approval. After the customer approves or
cancels the payment, Paypal will either call the returnUrl or cancelUrl you provided depending on
what action the customer performs.

def execute(){

    String clientId = grailsApplication.config.paypal.clientId
    String clientSecret = grailsApplication.config.paypal.clientSecret
    String endpoint = grailsApplication.config.paypal.endpoint
    Map sdkConfig = [:] //= grailsApplication.config.paypal.sdkConfig//['mode':'live']
    //sdkConfig.put("grant-type","client_credentials")
    sdkConfig.put(Constants.CLIENT_ID,clientId)
    sdkConfig.put(Constants.CLIENT_SECRET,clientSecret)
    sdkConfig.put(Constants.ENDPOINT,endpoint)
    def accessToken = paypalService.getAccessToken(clientId,clientSecret,sdkConfig)
    def apiContext = paypalService.getAPIContext(accessToken,sdkConfig)
    //the paypal website will add params to the call to your app. Eg. PayerId, PaymentId
    // you will use the params to 'execute' the payment
    def paypalPayment = paypalService.createPaymentExecution(['paymentId':params.paymentId,'payerId':params?.PayerID],apiContext)

    JsonSlurper slurper = new JsonSlurper()
    def map = slurper.parseText(paypalPayment.toString())

    redirect(url:"to your url")
}

Normally, if the 'createPaymentExecution' call to paypalService is successful, you will receive the payment. You may also need to perform other actions like storing details of the transaction on your db, marking your customer invoice as 'paid', etc.

Paypal provides a dashboard that you can use to monitor all transactions.


The grails paypal-payments plugin provides other features that developers can combine in order to fulfill app requirements. eg. Payouts, etc. 

That's it. Easy as pie. Go ahead and explore. . . .

You may use the comments box to share your thoughts. 


Thursday, September 17, 2015

Grails and Hibernate: Saving emoticons ( unicode ) to MYSQL.



We built a Grails application that uses a mail plugin, to send and retrieve emails from mail servers. The plugin employed a scheduled job to retrieve and store the emails on a MySql database table.

Initially everything worked as specified until recently when I noticed that some mails were not saved to MYSQL. The error message from MYSQL was 'incorrect string value' bla bla bla....

On close examination, I noticed that the troublesome emails contained emoticons (unicode characters). The unicode characters were responsible for my troubles :(

The issue


I dug through the MySQL documentation and learnt that my MySQL instance was having a hard time accepting those unicode characters because the database encoding was set to "utf-8".

Solution

To solve the problem required a change of the mysql default encoding from "utf-8" to "utf8mb4".

Note:We used a MySQL version of 5.6.13 but this solution will also apply to MySQL version 5.5 and higher.

If you are starting out with a fresh MySQL schema, all you need to do is specify the encoding in your schema creation script.


CREATE SCHEMA `your_db` DEFAULT CHARACTER SET utf8mb4 ; 



Then jump to Step 4.

For everyone trying to convert an existing db schema or table encoding from "utf-8" to "utf8mb4" please follow the steps below.

Step 1. Backup your database. 

The importance of creating backups cannot be overemphasized. Please create a database backup before embarking on this task.
  
To learn how to backup your database, please see MySQL Backup methods.

Step 2 : Upgrade your version of MySQL ( optional )

At this point, I assume you have performed a backup of your database. If your MySQL db version is below 5.5.x please upgrade your version to the latest available database ( newer is supposed to be better right? ) .

I am currently using MySQL version 5.6.13

Step 3 : Write schema charset conversion scripts

Run the following scripts for MySQL. If you do not know how to run MySQL scripts please contact your database administrator for help :) 


# To convert db:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;

#To convert db tables
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

#To convert table columns
ALTER TABLE table_name CHANGE column_name column_name VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;


Please do not copy and paste the code as is. You need to insert the appropriate properties/attributes which depend on your database name, table name, column type, etc.

In my case, I converted the database, and converted the important tables ;)


Step 4 Modify client and server connection char set


Locate your configuration file for MySQL; On a Windows machine, I found one at

C:/programData/MySQL/MySQL Server 5.6/my.ini

On a Linux/Unix machine, the location of the configuration file may depend on your MySQL installation directory and some other factors.

For simplicity I found my configuration file at //PathToMySQL/my.cnf

Open the file, and make sure property "character-set-server" is set to utf8mb4

I pasted a snippet from my config file below 

# The default character set that will be used when a new schema or table is
# created and no character set is defined
character-set-server = utf8mb4

Step 5 : Configure grails/hibernate

If you use Hibernate, then you need to update your configuration file. In my case, I use Grails, my configuration file is located at //PathToGrailsApplication/grails-app/conf/DataSource.groovy.


Open the file and look for the hibernate block. It will look like this

hibernate{....}


If you do not already have it then feel free to add it.

Inside the hibernate block, add the following.

hibernate{

    connection.charSet = 'utf8mb4'
    connection.characterEncoding='utf8mb4'
    connection.useUnicode=true
}


Save the DataSource.groovy file.

Done




Wednesday, August 12, 2015

How to upgrade Grails 2 applications and plugins to Grails 3

How to upgrade Grails 2 applications and plugins to work with Grails 3


 The Grails framework is a web framework for developing dynamic websites. It has been recently upgraded to version 3.0 and the upgrade has introduced breaking changes. According to the grails documentation, Grails 3.0 is a complete ground up rewrite of Grails and introduces new concepts and components for many parts of the framework.

The 'rewrite' part means that a lot has changed, including class packages.

This means that your plugin will break if it imports any of the Grails API packages that have been moved.

Although the Grails documentation provides a guide for upgrading applications and plugins, it does not provide information about all the Grails API package changes.


To save software development time, please find some of the package changes below.

Grails 2 Grails 3
org.codehaus.groovy.grails.web.servlet.mvc.GrailsParameterMap grails.web.servlet.mvc.GrailsParameterMap
org.codehaus.groovy.grails.web.servlet.GrailsApplicationAttributes org.grails.web.util.GrailsApplicationAttributes
org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequest org.grails.web.servlet.mvc.GrailsWebRequest
org.codehaus.groovy.grails.web.mapping.UrlMappingInfo grails.web.mapping.UrlMappingInfo
org.codehaus.groovy.grails.plugins.web.api.ResponseMimeTypesApi org.grails.web.mime.HttpServletResponseExtension
org.codehaus.groovy.grails.commons.GrailsApplication grails.core.GrailsApplication
org.codehaus.groovy.grails.web.mapping.DefaultUrlMappingsHolder org.grails.web.mapping.DefaultUrlMappingsHolder
org.codehaus.groovy.grails.commons.DefaultGrailsControllerClass org.grails.core.DefaultGrailsControllerClass
org.codehaus.groovy.grails.web.mapping.DefaultUrlMappingEvaluator org.grails.web.mapping.DefaultUrlMappingEvaluator
org.codehaus.groovy.grails.commons.DefaultGrailsApplication grails.core.DefaultGrailsApplication
org.codehaus.groovy.grails.commons.ClassPropertyFetcher org.grails.core.util.ClassPropertyFetcher
org.codehaus.groovy.grails.web.util.WebUtils org.grails.web.util.WebUtils
org.codehaus.groovy.grails.commons.ControllerArtefactHandler org.grails.core.artefact.ControllerArtefactHandler

Essentially, you need to change your package imports from Grails 2 plugins to the corresponding packages in Grails 3.

For more information about how to upgrade your grails application please see Upgrading your Grails application

Tuesday, August 4, 2015

Parsing and binding date formats from params in Grails.

"Grails is a powerful web framework for the Java platform aimed at multiplying developers' productivity" - grails.

We recently decided to rewrite some parts of our software ( written in grails )as web applications. 

The reason we made that decision is something I hope to write about soon. The task required us to send JSON strings (POST request) that contain datetime information along with other bits for data to the server. 

To bind a date string from the 'param' seamlessly ( without writing extra code to extract the date in your controller) you need to add the following to your grails config file ( config.groovy or application.yml ).

grails.databinding.dateFormats = [ "yyyy-MM-dd'T'HH:mm:ss.SSSXXX", "yyyy-MM-dd'T'HH:mm:ss.SSSZ" ,'yyyy-MM-dd HH:mm:ss.S','MMddyyyy' ]


With the above configuration in place, there was no need to extract the date and bind it manually to the domain class or command object. 

Sunday, December 14, 2014

Insights on Systems Dynamics & Its Relevance in Sub-Sahara Africa (SSA)

Dr. Mathias M. Fonkam, Damian Feese, Felicia Ikwu, Ama Oruomabo, Omasirichukwu Udeinya
1.      Introduction 
Systems dynamics is an offshoot or specialization of systems thinking that emphasizes the special role of computer simulation for understanding the dynamic complexity that is an intrinsic part of most management type problems and finding leverage points within the system from which fundamental, long lasting interventions might be effected with a hope to improving the problem situation. Both represent a paradigm shift from the traditional scientific method. The scientific approach to complexity is reductionist by nature since it prescribes breaking complex problems down into their parts with the hope that an understanding of those parts, or indeed a solution to the sub-problems they represent, can be assembled for the whole. Like systems thinking,  systems dynamics is a holistic approach to complexity that emphasizes the importance of the connectedness between the component parts of the problem and the important relationship between this structure of a problem situation and its perceived response or behavior over time. Furthermore, Systems Dynamics argues that it is this structure or connectedness between the system influences which is responsible for the dynamic character and the emergent properties that are unique to the problem situation as a whole. Fundamental components of this system structure that account for much of its dynamic character and complexity are feedback loops – both positive and negative loops – and delays. Systems Dynamics recognizes the fundamental limitations of the human capacity for dealing with complexity, especially dynamic complexity and therefore extends the Systems Thinking modeling technique of causal loop diagrams with a more quantifiable technique of stock and flow diagrams which easily translate to a computer simulation model that can be used to gain better insight on dynamic complexity and to discover points of leverage within the system. From the pioneering works of the Systems Dynamics Group at MIT many common recurring system structures and corresponding patterns of behavior have emerged that characterize the majority of complex problem situations. Even more importantly, for each system archetype a point of leverage is shown, where a change in the system structure will provide a more fundamental and long lasting improvement. In each case, this point of leverage appears to be counter-intuitive or contrary to where our intuition will lead us to want to intervene to effect change in the system. Understanding these system archetypes and learning to recognize them in problem situations provides a rich conceptual toolset in the decision maker’s toolbox and to a whole new approach to tackling complexity and one that embraces the full dimensions of the problem at hand with correspondingly better results.

We believe that Systems Dynamics holds great promise for Sub-Sahara Africa where most will agree serious systemic problems exist at all levels especially within government, business and other social organizations. More than half a decade since its inception at MIT, adoption of Systems Dynamics has been rather slow even in the Western world – the US and Europe from reports by the MIT group. Poor adoption does not negate its relevance and power as an approach to complexity. We find very scant adoption within Sub-Sahara Africa, yet it is exactly within this region that the most profound impact can be had from its adoption. In this short paper we hope to highlight some of the key elements of this important cross-disciplinary subject, just enough to arouse and motivate interest in the subject. Systems dynamics forms the core of our INF 490 – Systems Thinking & Modeling for a Complex World – a course offering for senior students on our Information Systems program but one that we hope to extend across campus as an important cross-disciplinary course open to all majors. World-wide, the course is especially important for students within schools of business. In the next section, we outline what systems dynamics is, why it is becoming increasingly relevant today and some of its key elements & thinking skills. Section 3 describes the conceptual modeling technique of causal-loop diagrams used to capture and communicate the structure of problem situations encompassing feedback loops and delays and generic behavior over time graphs that characterize some of these structures. Section 4 discusses a number of recurring system archetypes that have been elaborated from the works of the MIT group using various examples. Understanding these archetypes alone will shed important insights into the systems dynamics approach and its power to handling complexity. Section 5 outlines some common systems dynamics principles embodied in everyday language with examples to drive home the message and the shift in thinking that is required Also discussed in that section are a number of what a notable author on the subject, Peter Senge [Ref], has termed the disabilities of the modern organization that most people in organizations will relate to. Section 6 is our conclusion with pointers to various initiatives we are putting up here at AUN to promote this important discipline within the SSA region.

2.      The Systems Dynamics Approach & It Relevance in a Global Economy & SSA Region
Systems dynamics was invented in 1956 by Jay Forrester  at MIT as an extension of the general field of systems thinking that emphasizes the relationship between structure and behavior and the important role a computer simulation model can play in shedding insight on system structure and helping identify or discover points of leverage to effect long lasting change that drives the system in the desired direction. According to Forrester, systems dynamics combines ideas from traditional management, control theory and computer simulation. Like the general field of systems thinking systems dynamics adopts a holistic view of a problem situation. It extends the graphical modeling technique of causal loop diagrams with a more quantifiable stock and flow model that can also more easily be translated into a computer simulation model.  A computer simulation model serves as an important laboratory for learning and experimentation to discover leverage points within the system that are often counter-intuitive and hence hard to find relying simply on our mental models. More quantifiable behavior over time (BOT) graphs of the system under study can also be derived from such simulation models which will serve as important communication vehicles. Besides these graphical notations, systems dynamics introduces a methodology with a set of important thinking skills to enhance adoption of the holistic view point that always emphasizes the whole but without losing sight of the trees – we learn to see both the forest and the trees. In outline the methodology has the following phases and corresponding thinking skills that are briefly explained:
1.      Specify Problem/Issue  – which seeks to clarify the what of the problem and ensure that we are addressing the right problem and using an appropriate set of tools. As systems dynamics is more relevant for complex, dynamic time-varying problems, some important relevant thinking skills to adopt in this phase include; (1). Dynamic thinking which ensures and enforces a dynamic viewpoint and regards the visible behavior or performance indicator as part of an ongoing process with a dynamic character and perhaps even changing patterns of behavior over time, (2) System as cause thinking which tries to make explicit influences believed to be part of the process, and about which some control can be exerted, pulling from the mental databases of the stake-holders and experts, in the field, and (3) Forest thinking which seeks scope the problem situation both in time and space to encompass the influences and permit to discover the exact behavior patterns that characterize their actual interactions.
2.      Construct a dynamic hypothesis or model – focuses on an explicit model using causal loop diagrams and/or stock and flow diagrams that encompasses the feedback loops and delays between the system influences and reference behavior patterns described by behavior over time (BOT) graphs deriving from the feedback loops. Thinking skills that can facilitate activities during this stage include; (1) Operational thinking where some regard is given to understanding the physics of the problem area, i.e. how do things work? (2) Closed-loop thinking  which emphasizes the key systems thinking view that dynamics is generated by feedback loops which can be re-enforcing (positive) or balancing (or negative) and inherent delays in the transmission of material or information (3) Quantitative thinking – focuses on including soft influences (human related such as motivation, loyalty and commitment) that are often important parts of the system structure but which can easily be ignored because of difficulty of measuring them. Quantitative thinking not only encourages inclusion of such variables but teaches how to introduce numeric units of scale for such quantities that permits their inclusion in a simulation model.
3.      Test hypothesis or model – Through rigorous testing we get to check the appropriateness of the model, correct any inadequacies and refine the model to be more in line with the reality it represents. The key thinking skill to employ during this phase is Scientific Thinking which is based on the scientific viewpoint that progress in science is marked by discarding falsehoods rather than ascertaining “truth”.  Adopting this view means we can easily discover the limitations and usefulness of our models. Rigorous testing will also help discover leverage points in the system where change can be introduced for the desired effect.
4.      Implement changes & communicate understanding – the final phase of the systems dynamics process regards translating the results of the modeling exercise to one on the real-life situation and begins with the learning and change in the mental model of the participants and human actors in the system.
The figure below illustrates two scenarios for managing a complex problem situation both based on using models. The one on the left is based entirely on employing mental models whereas the one on the right illustrate where the systems dynamics methodology comes in to refine and shed better insight on the problem structure and hence improve the mental model for a better outcome.



Forrester & his group at MIT have successfully employed the SD methodology to shed important insights and even make correct predictions on key issues impacting some of the top business organizations in the world [Refs], government organizations and even issues of a global scale [Refs] such as pollution, food supply, population, disease, depletion of non-renewable resources etc. In many of such problem situations, the approach has been seen to be superior to the traditional scientific method in terms of the improvements obtained and the learning engendered. Such global issues affect Africa in equal measure. In fact, in some cases such as disease the impact on the SSA region is disproportionately higher. The systems dynamics approach is needed now as never before due to the increasing inter-connectedness of global business and the complexity that comes with that and an increasing recognition of a common heritage for mankind.

3.      Systems Dynamics Models – Causal Loop Diagrams & Behavior-Over-Time Graphs

Many real systems are comprised of a large number of interconnected elements and exhibit very complex behavior as they evolve over time. Causal loop diagrams provide a powerful method of capturing the inter-connectedness of these elements and gaining some insights on their dynamic complexity. Causal loop diagrams show how the various elements in the system are connected by cause-and-effect relationships, the nature of each of which is depicted by the use of the symbols S (or + sign) and O (or – sign). S (or +) implies that the elements connected by a specific cause-and-effect relationship move in the same direction (as customer satisfaction increases, sales revenue increases); an O (or -) implies that they move in opposite directions (as the workload increases, my ability to cope decreases). Single, closed feedback loops are of two and only two types. A reinforcing or positive feedback loop is characterized by having an even number of Os around the complete loop (with zero counting as an even number). We often recognize these loops as virtuous or vicious circles, for they exhibit exponential growth or exponential decline. The same cause and effect structure can behave in both ways—which behavior actually takes place in practice depends on how the feedback loop is initiated, and whether or not an actively spinning loop is subject to a sudden external shock. A balancing or negative feedback loop is characterized by having an odd number of Os (negative signs) around the complete loop. These loops exhibit goal-seeking behavior, often toward an externally determined target or budget. Sometimes the approach to the goal is smooth, but if there are time delays associated with the feedback loop, they can exhibit overshoot and undershoot, causing the system to oscillate, possibly wildly. Most problem situations will be characterized by several interconnected feedback loops and depict patterns of behavior over time. However, at any moment in time the behavior of the system will reflect a dominant loop in the system which is either positive or negative,.

For any given problem situation, there is a proliferation of mental models. Different people have genuinely and sincerely held different views on how the world works. CLD provide the opportunity to shine a perspective light on these mental models, share mental models and improve on their content through team learning, simulation, refining and critiquing the explicit models. Figure below shows a CLD typical for most business scenarios and comprising 2 feedback loops….

4.      System Archetypes, their BOTs & Points of Leverage
A number of system archetypes have been identified and documented by the MIT Systems dynamics group. These are recurring structures and behavior patterns for many complex problem situations. Points of leverage for these archetypes are often counter-intuitive which explains past failures of intervention efforts. Understanding these archetypes and learning to recognize them in similar problem situations will therefore shed a perceptive light on the correct points of leverage within the system and hence avoid similar traps within the system that appear suggest themselves as leverage points. We do not cover all the available archetypes to date. This is an ongoing area of research. In each case we name and briefly document the archetype, then sketch a causal loop diagram for its structure and behavior over time graph. Then we discuss leverage points within the system that have been shown to yield fundamental and long lasting change.

4.1.  Fixes that Fail – often phrased as “Today’s problems come from yesterday’s solutions”
This archetype describes the familiar situation where interventions to improve a problem situation lead over time to a worsening of the problem. Initially the fix appears to work, but then after a delay, which may be weeks or months later, the problem resurfaces, often with greater force.  Its causal loop model is shown below:



A fixes that fail structure consists of a balancing loop which is intended to achieve a particular result, yet the result is foiled by an insidious reinforcing loop whose effect is delayed as portrayed by the clock. These two loops interact in such a way that the desired result initially produced by the balancing loop is, after some delay, negated by the actions of the reinforcing loop. A BOT graph of this will show slight jerks of improvement at each intervention point but generally a worsening of the problem over time. An effective strategy to dealing with this archetype is advanced planning and thinking through all the unintended consequences of each action, their own consequences and so on until you can assess the full impact of a decision.

4.2. Shifting the burden – often described as procrastination
 This archetype captures the structure of a common human behavior pattern which relates to our tendency to deal with the easy, the obvious and urgent before one is forced to confront the difficult, the ambiguous and the important. Often we put off difficult decisions in the hope that they will go away. A good example is using alcohol or drugs to suppress personal difficulties or depression. A Shifting the Burden structure is composed of two balancing loops and a reinforcing loop. It is a very annoying structure because the two balancing loops act as a single reinforcing loop driving the situation in the same direction as the reinforcing loop. Both structures support the movement of the system in a direction generally other than the one desired. The behavior pattern is very similar to that of Fixes that fail model. Here however, the fundamental solution is masked both by its inherent time delay, the impact of the symptom and side-effects of employing the symptomatic solutions.



A good example of this archetype is Perpetuation by Self-deception shown below:



The most effective strategy for dealing with a Shifting the Burden structure is an employment of the symptomatic solution AND development of the fundamental solution. Thus one resolves the immediate problem and works to ensure that it doesn't return.

4.3. Limits to Growth (or Success)
A Limits to Growth structure consists of a Reinforcing Loop, the growth of which, after some success, is offset by a action of a Balancing Loop.  This model captures the structure behind the growth and leveling off or decline behavior of most businesses, cities and other human activities that start off with growth or success. Its CLD is shown below:


In the model the focus tends to concentrate on the positive loop since it drives the desired result. The re-inforcing loop normally dominates for some time too but eventually its dominance is offset and may even be reversed by the negative loop. However, emphasis may continue to be exerted on the positive loop for corrective action. The behavior pattern for this model is an S-shaped growth or even an overshoot and collapse.The best defense is a good offense. If  there is a Reinforcing Loop operating start looking for what is going to become a limiting factor, and remove it before it even has a chance to create a substantial impact on results.

4.4. Drifting Goals – lowering the bar
In this model a gap between a goal and the current reality is resolved by taking corrective action (the normal path) or lowering the goal (the drift). Lowering the goal usually closes the gap immediately, whereas corrective actions usually take time and more effort.  A drifting goals structure is composed of two balancing loops which interact in such a way that the activity of one loop actually undermines the intended balance the other loop seeks to achieve. Consider the following example in which I set out to pursue something I want. The CLD model follows.



There is only one real effective way to deal with this structure and that is to disconnect the feedback from pressure to settle for less to what I want so it can no longer subtract from what I want. Either you want it or you don't, and indecision is your problem then see Indecision.

4.5. Escalation
An escalation structure is composed of two balancing loops which interact in such a way as to create a single reinforcing loop. The action of each loop provides the basis for increased action by the other loop and the real foundation for this is insecurity resulting in competition. Examples of problem situations with this structure include the heating up of an argument between two persons, the US/Soviet arms race or how urgency begets urgency. In the business environment price wars between competitors provide a good example. Its CLD model is as follows.



The 2 balancing loops interact to produce an overall re-inforcing loop that explains the escalation as shown below.


A good leverage point is to engage both parties in such a way that they begin to see the value in collaboration or to completely disconnect them into two separate loops that are self-determined.

4.6. Tragedy of the commons – often described as “The all for one and none for all”
In this model a common resource is over-exploited competitively by two or more parties generating an overall activity that negates the gain for everyone. The CLD for this is shown below. Examples of this structure abound in business and social organizations sharing a common but limited resource known to both parties and on which their success depends. Welfare hand-outs creates such systems too.


The most effective strategy for dealing with this structure is to wire in feedback paths from A's results and B's results to the resource limit so as A and B use resources their results promotes the availability of additional resources.

4.7 Success to the successful
A Success to the Successful structure consists of two reinforcing structures which interact in such a way as for create a single reinforcing structure. Consider a situation where there are two project managers, Jane and Tom, responsible for managing similar projects.


Their manager, Sarah, has a fixed amount of resources which she allocates to their projects. Initially both projects are progressing equally well. Then, for some reason, Sarah chooses to allocate more resources to Jane's project than to Tom's.  The structure can be redrawn as a single re-inforcing structure as below.



A good example of this that can easily be seen in an academic context is shown below.



There are actually two strategies for dealing with a Success to the Successful situation.
1.      Identify the resource(s) being unequally distributed and balance the distribution.
2.      Disconnect the two reinforcing structures so they are not dependent on the allocation of shared resource(s).


5. Organizational Disabilities & Systems Dynamics Principles in Everyday Language
In his best-seller, “The Fifth Discipline The Art and Practice of the Learning Organization”, Peter Senge identified a number of what he describes as learning disabilities that characterize the majority of organizations which he classifies as ordinary organizations and typifying the majority of organizations today. In Senge’s view, and many will agree with him, “the only sustainable competitive advantage is the rate at which organizations learn”. These disabilities make it very hard to see the big picture and hence to make interventions that have a lasting impact on the system as a whole. We only very briefly introduce with a comment or two but invite the interested reader to refer to Senge’s book for more insight on each disability.
5.1.   I Am My Position – we focus so narrowly on our jobs that we confuse them with our identify and see ourselves incapable of intervening outside the boundaries of our jobs
5.2    The Enemy Is Out There – a by-product of the I am my position view that makes it hard for us to think systemically so we develop a propensity to always look for someone else to blame when things go wrong rather than see ourselves as part of the whole and the problem. Out there and in here are often part of the same system.
5.3    The Illusion Of Taking Charge –proactiveness is in vogue in management cycles yet all too often it translates into reactiveness and fighting the enemy out there rather than looking inwards to see how we contribute to our own problems.
5.4    The Fixation on Events – this tendency distract us from seeing the longer term patterns of change that lie behind events and from understanding the causes of those patterns.
5.5    The Parable Of The Boiled Frog – coined after several systems studies of corporate failure revealed the same results that maladaptation to gradually building threats to their survival is rather pervasive. Many corporations are better at reacting to events than detecting build up in threats that threaten their survival, very similar to the nature of frogs.
5.6    The Delusion Of Learning From Experience – learning from experience remains the most powerful way of learning but when our actions have consequences beyond our learning horizon, it becomes impossible to learn from direct experience.
5.7    The Myth Of The Management Team – all too often, teams in business tend to spend their time fighting for turf
Recognizing the constraining actions of these learning disabilities Senge and his group have established a set of laws of the fifth discipline which are essentially common sayings that capture an important tenet or viewpoint of systems thinking. We will simply list them here as their meanings are well known.
a)      Today’s problems come from yesterday’s solutions.
b)      The harder you push the harder the system pushes back
c)      Behavior grows better before it grows worse
d)      The easy way out usually leads back in
e)      The cure can be worse than the disease
f)        Faster is slower
g)      Cause and effect are not closely related in time and space
h)      Small changes can produce the biggest results but the areas of highest leverage are often the least obvious
i)        You can have your cake and eat it too but not at once
j)        Dividing an elephant in half does not produce two small elephants
k)      There is no blame

6. Conclusions

References
  1. David Bayless & Don Greer, “An Introduction to Systems Dynamics”, The Venture Dynamics Group, 2004.
  2. P. Senge, “The Fifth Discipline, The Art & Practice of the Learning Organization”, Bantam Doubleday Dell Publishing Group, Inc.,1994.
  3. John D. Sterman, “Business Dynamics: Systems thinking and modeling for a complex world” , McGraw-Hill,2000.
  4. Vensim – A Systems Dynamics Simulation package: http://www.vensim.com/
  5. Thomas Binder et al, “Developing Systems Dynamics Models from Causal Loop Diagrams”,
  6. George P. Richardson, “Insightful Little Models”, College of Public Affairs & Policy, University of Albany, NY.
  7. Meadows, D., Meadows, D., Randers, J., and Behrens, W. 1972. “Limits to Growth”. New York: Universal Books. 
  8. Denis Sherwood, “Seeing the Forest for the Trees, A Manager’s Guide to Applying Systems Thinking”, Nicholas Brealey Pub., London, 2002
  9. Barry Richmond, “The Thinking in Systems Thinking – Seven Essential Skills”, Pegasus Communications, Inc, www.pegasuscom.com.
  10. John Boardman & Brian Sauser, “Systems Thinking, Coping with 21st Century Problems, CRC Press, ISBN-13: 978-1-4200-5491-0
  11. http://www.systems-thinking.org/theWay/theWay.htm