This post continues my practice of tidying up past posts to make the material more accessible.
Most professionals use case studies. However, the practice is especially important in training. For that reason, I ran an earlier short series on the creation and use of case studies. You will find the introductory post here.
Useful post by Tony Karrer on elearning Technology on the optimum length of an e-learning course. Tony's key point is that it should be as short as you can make it.
This has certainly been my own experience. Most on-line courses that I have seen within organisations focus on providing information. An example might be an induction course. Too often, the developers include a range of material that they think might be useful.
Bear in mind that the old rule of thumb used to be that a web page should contain at maximum 60 per cent of the material on an equivalent printed page simply because web pages are harder to read than the printed equivalent. So prune, prune, prune.
In my last post in the Towards a Discipline of Practice - Evidence Based Medicine series I looked at some of the problems associated with evidence based medicine. In my second post in the series I quoted the definition of evidence based medicine supplied by the Centre for Evidence Based Medicine:
Evidence-based medicine (EBM) is the integration of best research evidence with clinical expertise and patient values ... When these three elements are integrated, clinicians and patients form a diagnostic and therapeutic alliance which optimizes clinical outcomes and quality of life. Centre for Evidence Based Medicine
I suggested that this quote captured four key elements relevant to all professions and professionals.
The first element was research evidence, essentially what works and why.
The second element was professional expertise, our capacity to understand and apply our professional knowledge in the circumstances of the particular case.
The third element was the patient or client, each with their specific attitudes and needs.
The final element was the integration of the first three elements - the diagnostic and therapeutic alliance - to provide the solution that best meets client needs.
Using these four elements as a framework, I now plan over forthcoming posts to extend my analysis by looking at some of the lessons from evidence based medicine for professional practice in general.
In my last post on evidence based medicine, I suggested that professional training focused on the transfer of existing knowledge and skills to the new professional, knowledge and skills that the professional then applied. The professional subsequently built on this base through practice, at the same time using various professional development activities to try to keep in touch with new developments.
I then posed the question what happens if that existing knowledge base is in fact wrong, looking briefly at the reasons why this proved to be the case in medicine, a discovery that had led to the development of evidence based medicine. I concluded that, as with any other approach, evidence based medicine had its own methodological problems. However, it also had important lessons for other fields of professional practice.
The Quality Movement and Quantification
In a series of posts on my personal blog I explored some of the changes that had taken place in public administration since the war, looking at the influences on those changes.
In one of those posts I looked in part at the way in which standards, the Quality Movement and the importance of measurement had become major global influences. I also suggested that the outcomes here had not always been positive.
Evidence based medicine forms part of the global standards and quantification revolution and suffers from some of the same weaknesses. These weaknesses need to be recognised.
Problems with Evidence Based Medicine: Perception Bias
The first problem can be called simply perception bias.
In another post on my personal blog on science and political correctness I looked at ways in which dominant views acted to exclude alternatives.
Evidence based medicine is neither value nor perception free. The questions selected for test and evaluation, a process that can be very expensive, are influenced by prevailing views. Valuable alternatives may be excluded simply because they fall outside conventional wisdom. As evidence based medicine becomes the dominant mode, the effect may, as it has been in other areas, to actually narrow fields of investigation and action.
This links to a second problem, one that I have discussed before, the tendency for all professions to see answers within a frame or world view set by their profession.
A lawyer will give you a legal answer to a problem, a doctor a medical answer. If you have a back problem and see a surgeon, he/she is likely to think about surgical solutions. Go to a chiropractor with the same problem and he/she is likely to recommend spinal manipulation. So professional background helps determine the way the problem is defined, the solution applied.
This flows through into the application of evidence based approaches because the things tested are generally set within the frame of the tester. So evidence based medicine focuses on the efficacy of medical treatment and may leave non-medical alternatives aside.
Problems with Evidence Based Medicine: Causation
As part of my history honours year in my first degree I studied philosophy of history under Ted Tapp. Ted was a reflective man who required us to think about, to debate, the philosophical underpinnings of science and scientific method.
One core problem was the difference between correlation (a and b) as compared to causation (if a then b).
This problem applies in evidence based approaches. Just because a study shows an apparently strong relationship between a treatment and positive patient outcomes does not necessarily say anything about the causal relationship between the two. This has to be deduced and further tested.
Problems with Evidence Based Medicine: Problems of Epidemiological Studies
The problem of correlation vs causation links to another group of problems with evidence based medicine.
By its nature, evidence based medicine deals with large groups, populations.
As trials become larger and more complex, it becomes more difficult in statistical terms to establish significant relationships, to separate the effects of different variables.
This creates another problem, the establishment of a clear relationship between the outcomes of trials at population level and subsequent application at individual level.
As the Wikipedia article notes:
Critics of EBM say lack of evidence and lack of benefit are not the same, and that the more data are pooled and aggregated, the more difficult it is to compare the patients in the studies with the patient in front of the doctor — that is, EBM applies to populations, not necessarily to individuals.
This can create very real difficulties for individual clinicians, leading Tonelli to argue in The limits of evidence-based medicine that:
the knowledge gained from clinical research does not directly answer the primary clinical question of what is best for the patient at hand.
Tonelli concludes that proponents of evidence-based medicine discount the value of clinical experience (source Wikipedia).
Problems with Evidence Based Medicine: Impact of the Observer
Another problem with evidence based medicine, one often seen in all evidence based approaches, is the way the observer affects the observed. This happens at several levels.
The first problem is that the simple act of participation in the trial may have some and not clearly seen impact on individual outcomes. In medicine, this is usually managed by use of a control group using a placebo. The efficacy of the treatment is then measured by the difference in outcomes between the control group and those receiving the treatment.
A second linked problem is the impact on patient behaviour of the trial itself. By their nature, clinical trials are closely managed. This means that patient compliance with the treatment routine is likely to be high.
This need not hold in subsequent clinical use since ordinary patients are more likely to fail to follow treatment processes by, for example, failing to take medication exactly as prescribed. This means that actual patient outcomes may not be as good as the trial results.
Problems with Evidence Based Medicine: Limitations in Application
A further problem is that the most rigorous gold standard approaches dictated by evidence based medicine can only be applied in narrowly defined circumstances, leaving a range of medical approaches that have to be tested by less rigorous means.
This should not matter so long as the limitations are recognised. In practice, it risks introducing two distinct distortions into the medicine and the health system.
The first is the risk that investigation may be biased towards those things that can be measured through more rigorous techniques, reducing thought and investigation in areas less amenable to measurement.
The second related risk is that treatment itself may become biased.
At clinician level, this links back to my earlier point about perception bias. Doctors trained in evidence based medicine may, consciously or unconsciously, come to focus in treatment terms on those things that can be measured, ruling out other less easily measured options.
This tendency may be reinforced by actions from those managing or funding the provision of health care services who may refuse to allow/pay for certain types of services notwithstanding the views of individual clinicians.
Next Post
I have focused in this post on problems associated with evidence based medicine. In my next post I will look at the lessons of evidence based medicine for other professions.
Previous Posts in the "Towards a Discipline of Practice" Series
Evidence-based medicine (EBM) is the integration of best research evidence with clinical expertise and patient values ... When these three elements are integrated, clinicians and patients form a diagnostic and therapeutic alliance which optimizes clinical outcomes and quality of life. Centre for Evidence Based Medicine
While this quote focuses on medicine, it also captures four key elements relevant to all professions and professionals.
The first element is research evidence, essentially what works and why.
The second element is professional expertise, our capacity to understand and apply our professional knowledge in the circumstances of the particular case.
The third element is the patient or client, each with their specific attitudes and needs.
The final element is the integration of the first three elements - the diagnostic and therapeutic alliance - to provide the solution that best meets client needs.
In my brief introductory post on the case of evidence base medicine, I suggested that my instinctive first reaction on hearing about evidence based medicine was to say that's odd, I thought that all medicine was evidence based. I also suggested that this has proved, in fact, to be far from true.
Before continuing my analysis I should note that the Australian Broadcasting Corporation's Radio National has just completed a rather good two part series - Facing the Evidence - on evidence based medicine. The first part is already available in transcript and pod cast. The second will be available shortly.
Returning to my theme, when you look as I did in my last post at the standard way all professionals are trained, you can see that that training focuses on the transfer of existing knowledge and skills to the new professional, knowledge and skills that that professional then applies. The professional then builds on this base through practice, at the same time using various professional development activities to try to keep in touch with new developments.
But what happens if that existing knowledge base is in fact wrong? How might this arise? To quote from the first part of the ABC program:
Every day doctors and other health professionals use treatments that are harmful, or fail to use therapies that have been proven to work. In the US there is so much medical error that Congress has directed the Institute of Medicine to develop a strategy to improve the quality of care. In its initial report the Institute noted perhaps as many as 100,000 Americans die every year from medical errors, including the use of inappropriate treatments. That's much more than from car accidents, breast cancer or AIDS. Many more suffer side effects and unnecessary costs
How could this happen?
Part of the problem here lies simply in the placebo effect, that fact that patients respond just to the fact of treatment. So the treatment appears to work, thus supporting the original judgment.
Part of the problem also lies in the fact that individual outcomes can be affected by so many variables and over a considerable time horizon so that the fact of adverse outcomes may not be clear in an individual case or, if clear, may be due to a whole variety of factors external to the treatment itself. There is a linked issue here that relates to the size of the population.
Given that individual outcomes vary greatly, the fact that there is a problem and its scale and scope may only become clear if you look at a population as a whole. That is, the individual professional may have no easy way of detecting the problem in his/her individual cases.
A further problem lies in the nature of the models used.
All professions use models to try to explain a complex world. In economics, for example, models are a common method used to analyse economic behaviour and to suggest possible responses at firm and public policy levels. In the case of medicine, biological models are common.
The problem with all models is that they involve selection of key variables and the specification of relations between those variables. Get either wrong, and outcomes may be very different from those projected by the model.
Doctors have always been concerned about adverse or unexpected outcomes.
In 1972 Professor Archie Cochrane, a Scottish epidemiologist published Effectiveness and Efficiency: Random Reflections on Health Services. This plus Cochrane's subsequent advocacy caused increasing acceptance of the concepts behind evidence-based practice. Cochrane's work was honoured through the naming of centres of evidence-based medical research — Cochrane Centres — and an international organisation, the Cochrane Collaboration.
The explicit methodologies used to determine "best evidence" were then largely established by the McMaster University research group led by David Sackett and Gordon Guyatt. According to the Wikipedia article on evidence based medicine, the term "evidence-based medicine" itself first appeared in the medical literature in 1992 in a paper by Guyatt et al. (Guyatt G, Cairns J, Churchill D, et al. [‘Evidence-Based Medicine Working Group’] "Evidence-based medicine. A new approach to teaching the practice of medicine." JAMA 1992;268:2420-5. PMID 1404801)
From this point, the spread of the concept and its subsequent inclusion in professional training was rapid.
As with any other approach, evidence based medicine has its own methodological problems. However, it also has important lessons for other fields of professional practice.
Previous Posts in the "Towards a Discipline of Practice" Series
Evidence-based medicine (EBM) is an attempt to more uniformly apply the standards of evidence gained from the scientific method, to certain aspects of medical practice. Specifically, EBM seeks to apply judgements about the inductive quality of evidence, to those aspects of medicine which depend on rational assessments of risks and benefits of treatments (including lack of treatment). According to the Centre for Evidence-Based Medicine,"Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients."[1] Cited from Wikpedia.
In this post I want to continue my discussion on the development of a discipline of practice, a discipline focused on the way professionals practice their profession, by introducing in a preliminary way the case of evidence based medicine.
I first came across the concept in 1998 when I started as CEO of the Royal Australian (now Australian and New Zealand) College of Ophthalmologists. My instinctive first reaction was to say that's odd, I thought that all medicine was evidence based. In fact, that's far from true.
To understand this, we need to look at the way in which doctors are trained, as well as the way in which new medical approaches develop.
As with all professions, training starts with the previous body of knowledge relevant to that practice. For practical reasons, much of this has to be taken for granted. The trainee professional simply has to learn those elements required to begin practice.
The trainee then has to learn to apply that knowledge in practice. In the case of medicine this is done especially in hospitals working under the supervision of a qualified doctor who passes his/her knowledge and experience onto the trainee while checking their application. Again, the trainee is acquiring the current wisdom.
These broad processes continue throughout a professional training that can, in the case of medical specialists, extend as long twelve or thirteen years.
Once the doctor begins practice, he/she continues to learn from experience with patients. Doctors also learn through contact with other doctors and are expected to maintain a program of continuing professional education to keep them in touch with latest developments. Similar approaches and programs apply in other professional areas.
None of this will seem strange to any professional from any discipline. Yet in the case of medicine the process has proved to be seriously flawed. Evidence based medicine attempts to address these flaws.
Previous Posts in this Series
I did not comment on the Learning Circuits big question for November - Are ISD / ADDIE / HPT relevant in a world of rapid elearning, faster time-to-performance, and informal learning? - simply because I add little to add.I commend the November debate to those interested as a simple and quick way of coming up to speed on the issues involved.
The December big question (more accurately questions) is more open ended:
I cannot resist answering from a personal perspective.
What will I remember most about 2006?
For me, 2006 has been the year of the blog.
I have always written as a way of clarifying and explaining ideas. However, in recent years this writing has been generally client or Group member(s) specific.
I was aware of blogs and blogging as a way of expressing ideas, but this was not central to my work or life. Then Sandra Welsman, one of my Ndarala colleagues, pointed me to the Is training snake oil debate? on the Learning Circuits blog. Things snowballed from there.
We prepared a paper on the snake oil debate (link above), followed by a case study on the use of blogs as a communications device within specialist medical colleges. To further the learning process, I then began experimenting in April (was it only April?) with my own and Group blogs, looking further at the practical issues involved with blogging.
In turn, this led to the use of blogs as a publishing device for making some of our material more readily available to a broader audience, as well as a platform for expressing comments and ideas. Since then, I have published several hundred thousand words combining both existing and new material.
I have always combined curiosity (I like to know the how and why) with broad interests. When I first started blogging there was a period while I felt my way. As the quantity of posts grew, I found that the material published on the various blogs was overlapping and informing other material.
For example, the analysis I did on demographic change on my personal blog was intended to help me understand the impact of demography on social, economic and political change. But this then fed back into writing on this blog. In similar vein, I have re-published past material on the Ndarala Group blog to provide back-up for some of the stories written on this blog.
One important feature of the year has been the way in which blogging has put me in touch with other professionals, further informing my own thinking. This has been especially important on the learning and development side.
I believe very strongly that improved people management is critical to improved performance within professional services. This is reflected in the weight that I have placed on people management issues in my posts. I also believe that improved people management is too important just to be left to the HR professionals, that it needs to be implemented across firms.
I also believe that good training is central to improved people management. Here I have drawn heavily on the knowledge of others and especially those outside Australia to inform my own training perspective. I do not necessarily agree with them, I have real concerns about the US approach to education and training, but the insights have always been helpful.
In all this, I suppose that one of my conclusions from the year is the need for learning and development professionals to come out of the ghetto that they seem to have imposed upon themselves. As someone looking at education and training in part from an external management perspective, it sometimes seems to me that I have a higher opinion of learning and development than do those working exclusively in the field.
Looking Forward to 2007
In looking forward to 2007, I would join the questions of predictions and challenges together.
Many of the younger Australian professionals have never seen an economic downturn. They live in a world of constant market expansion. I have and know what its like.
Professional services is what economists call a leading indicator in that many areas involve discretionary spend that firms can and do defer as economic conditions tighten.
In 1990 the Australian marketplace for professional services broadly defined collapsed, with total fees falling by over a third in less than nine months. In the case of my own firm, a start up that had grown rapidly over the previous eighteen months, we started 1990 with monthly fees of $72,000, monthly expenses of $62,000. Over the next three months, monthly fees dropped to just $28,000. In so doing, we went from comfortably profitable to financial bleeding.
I make this point because both the world and Australian economy are increasingly uncertain.
In Australia, the growth shown in the GDP figures is coming from the mining states. In the latest quarterly figures NSW, the biggest state in economic terms and as a professional services market, actually showed a GDP decline. If, as seems quite possible, this is repeated in the December quarter, NSW will be technically in recession.
So economic conditions pose the first challenge.
The second challenge comes from demographic change.
I have suggested before that the implications of demographic change pose the key strategic challenge for professional services firms in western economies over the next decade. I have also noted my frustration at my failure to get this message across.
Now it may be that economic downturn will ease the short term problem by reducing demand for professional labour. But this will be at best a short term easing.
The longer term problem can be stated quite simply.
The numbers in the entry level age cohorts are stagnant to declining. High university fees are reducing higher education participation rates in many countries. So the people that professional services firms have traditionally relied upon are simply not there.
In the first instance, the stronger firms will survive by eating up the people market for the rest, if at a higher price. We can already see this.
In countries like Canada and Australia, areas outside the main metro centres are already suffering from an aging professional structure. This will go critical over the next decade as people retire. At country level, some countries (Germany is an example) are already suffering net migration decline among younger professionals attracted to better prospects elsewhere.
This poses challenges at both profession wide and firm level.
At professional level, the inability of individual professions to meet needs will lead to continued erosion in professional divides. We can already see this. In the absence of sufficient doctors, other professional groups such as registered nurses are being given responsibilities such as prescribing rights previously the preserve of medicos. In law, areas of law such as conveyancing are now going to non-lawyers.
At firm level, firms will be squeezed between limited staff on one side, new competition on the other. Again, the bigger who can afford to pay more will be able to cherry pick among the rest.
In all this, improved people management will be critical to a firm's ability to attract, retain and upgrade its people. Failure here means longer term death.
Linking this back to learning and development.
The challenge for all education and training professionals is going to be to find the best way for both professional services as a sector and Government to respond to these challenges. We cannot help all firms, only those prepared to help themselves. But for those firms who are prepared to change, we are going to need to be able to offer new solutions.
1. How is this case to be used?
2. What questions will generate the analysis and/or decision-making users will complete?
3. What analysis do I want the participants to do that will further the course objective(s)?
4. What are the questions (& sequence) I plan to ask?
5. What process options will I use?
Revision
What adjustments need to be made to the case document prior to distribution to readers?
1. How might the case be re-written so that I can teach it to achieve my objectives and the determined learner outcomes?
2. What is unclear/confusing/incomplete?
3. What needs to be added/taken out?
4. Do any perspectives need to be further developed?
5. What structural changes are needed? e.g., what decision point(s) do I want? where? Should the case break into parts (A) (B) ... ? What activities/timing are relevant for each segment?
Note on Copyright
The case study material is drawn from an Ndarala Group Guide prepared for the use of member professionals and clients. It is copyright Ndarala but may be copied with due acknowledgment. Incorporated material drawn from other sources should also be acknowledged.
Previous Posts
In the last post I provided a general introduction to the creation and use of case studies. This and the the following post look in detail at the development and use of case studies for training purposes.
The material is directed at both training professionals and other professionals who may wish to use their material for training purposes or who hire trainers.
The material assumes that you have already looked in at least broad terms at the questions posed in the previous section.
The preparation of the case study starts with the questions to be answered. The case may already exist as a general case study, in which case it is being modified for training purposes, or it may be prepared specifically for training purposes.
The writer of the case study should answer the following questions. This person can be either the ‘expert’ who has all the information required at their fingertips, or a researcher who collects and drafts the case document. The questions are drawn from John Boehrer (1998).
1. What is the hard, unifying question?
2. What information do we need to address it?
3. What can I ask users to analyse? decide?
4. What decision process does the case give the reader to focus upon?
5. What questions can I ask about it?
6. What detailed narrative does the reader need to focus on it the important issues?
7. What information needs to be developed?
8. What will make the story work as a case? What conflict does it present?
9. What will make the case livelier? What quotes or details? What will make the case more personal?
10. What will enable readers to identify with the actors and get into the case?
11. How do the actors themselves perceive the situation? Can they be quoted?
12. What factors/constraints do the case actors perceive to be impinging upon them?
13. Who else is involved? What is their perspective/outlook?
14. What development/structure would be useful?
15. What might different parts of the case focus upon?
16. What is the link between the specific decision focus and a larger question?
17. What is the issue context?
18. What special problem does writing/revising the case pose? Information? Access?
Having answered these questions, we can move to the preparation/modification of material. A basic structure here is set out below.
Case Study— a comprehensive example of the concepts one wishes to convey. Often the case incorporates materials that describe or simulate the example. In using a case study, we are moving from the concrete and specific to generalised abstract concepts and principles.
3. Case Summary
Note on Copyright
The case study material is drawn from an Ndarala Group Guide prepared for the use of member professionals and clients. It is copyright Ndarala but may be copied with due acknowledgment.
Previous Posts
