|
|
|
 |
15 July, 2018: Created
21 July, 2018: Embellished
Indian School of Development Management runs a programme in development management & leadership. As of the date of writing, the author is part of the academic working group tasked with design of the curriculum and related elements, one of which is assessment. It is difficult to get into details of the curriculum design at this point; however, the reader should get a jagged working sense from the article.
The author, additionally, had the good fortune to be associated with the execution of the programme in its first year. This discussion note gathers author's private reflections on what can be inferred from this year long experience as it relates to the assessment framework. It is private in the sense it reflects author's own unpacking of his thoughts: the content, by no means, is remarkable or original to speak of.
The value of this piece to a general reader maybe in the contrast it provides to the present post-graduate assessment paradigm.
DEVELOPING AN ASSESSMENT RUBRIC
Preface
The underlying objective of the ISDM Post Graduate Programme in Development Management is to aid, adapt, influence, but at the same time merge with, the practice of development management and leadership as it exists today. The operative term therefore is practice, and the accompanying academic technique chosen to suitably equip individuals is context-based learning.
A student who emerges from such a process may properly be termed a Reflective Practitioner. Why? Context-based learning equates to an ability to define the context outside of the 'I', observe it, understand its meaning, carefully infer from it, and thereby, arrive at an independent and collectively beneficial perspective; and having thus expended sufficient effort, at the very next moment, be quite willing and eager to challenge herself, and be challenged so as to repeat the process with greater finesse next time.
To begin with this demands considerable effort: but after a fashion, it becomes a habit, a natural way of making decisions and even carrying out one's other affairs. How does one then really acknowledge, appreciate, assess and appraise a Reflective Practitioner? This forms the heart of the question of assessment for the programme under consideration.
A convenient point of entry (to this question) lies in deconstructing the terms practice and reflection; and in the process, glean 'assessable' handles apposite for the frameworky nature of a formal rubric.
It goes without saying that more the 'passes' of deconstruction finer the gleaning. The first pass was probably made in midst of the first year of the programme. This, for all practical purpose, can be deemed to be a second pass: made with the foresight of doing it somewhat in advance, and with the hindsight of one year's worth of experience interacting with students.
More such passses are bound to follow. It is therefore prudent to not fixate on a particular schema or framework of assessment (adopted) for a given year as sacrosanct. This caution should dissuade attaching too much significance to a particular student's performance: whatever be the schema, one can never be sure it cuts a fair deal for all students concerned. Any schema thus compulsorily comes conjoined with the disclaimer of compassion.
An inability to fairly cover all dimensions implies too that the process of assessment will always retain a higher than usual element of subjectivity. Perhaps subjectivity is a term liable to be misconstrued: it maybe more in line to say when it comes to judging reflective practice, its veracity is tied more to the skill of the assessor than any rubric.
This means any discussion on assessment, besides the rubric of assessment, would benefit from debating the kind of persons who will assess, and the design of conversations that enable a student to bring her authentic self to the table. It is quite difficult to imagine an idiot-proof assessment method for a programme of this nature. Trying to do so will lead to detachment of the logic of assessment from the programme.
Relation to current assessment rubric
Having muttered these pleasantries, it should be noted this second pass is not a critique of the present framework (which looks at a student from four broad categories of Knowledge, Skill, Application and Personal Mastery). In going through a process of renewed thinking, it seeks to break apart the existing categories and stich the pieces together in a different pattern. This is done to bring the framework closer to what actually transpires with the student as she goes through her eight-odd terms, in addition to what actually happens during the process of assessment itself.
It is almost certain that those reading through the following paragraphs will come out with an altogether different embroidery from the same threads: the ladies have a slight natural edge on this, but just barely. This then is the express purpose of this document: to present a semi-finished product, and leave its moulding to others.
Deconstructing Practice
Practice is anchored in experience: created by dint of one's own efforts at reading, observation, listening, travel, conversation, companionship & collaboration with another. In other words, making the best of what circumstances can offer. The richer the ability to create, latch onto, record, and remember experiences more intricate is likely to be one's professional practice. It is foremost a question of intent and then of skills. Those who are disinclinded to engage, participate in and create experiences, despite all of their intellectual endowments may struggle to emerge as fruitful and authentic Reflective Practitioners.
For an academic institution, it is possible to engineer and design formative experiences into its academic calendar; such as what ISDM attempted in the first year (of its programme). There is always a possibility to improve their number and nature; and at some level, it may even be beneficial to keep tinkering with this part of curriculum design and not allow it to settle down. It is what lend's freshness and retains the programme's emergent nature.
From an assessment standpoint two observations beg notice: the experiences will certainly be designed and provided for, but what will be assessed is the ability of the student to sense their significance, accumulate them individually and as part of a group, and properly document them. The second, and strongly related, is the student's learnability most evident in how inclined she is to modify her habits on receipt of feedback.
The first step then is to be able to paint a student's record of participation, and changes in its pattern over the course of the curriculum. The best technique to arrive at this is through a conversation with the student herself. A typical such conversation will usually look to satisfy the assessor on following questions:
1. Has the student engaged with different elements of the programme in an even manner?
2. Has she shown an intent to actively make changes to her pattern of engagement? Did she take on elements she was previously shy of? How deep did she go in her areas of strength? How long was the time lag between being given a feedback and really internalising it?
3. Having thus engaged, what has been quality of her perceptions? What did she observe and what did she lose out on? Why? This aspect is dependent strongly on skills within the domain of personal mastery, of which most relevant is deep listening.
4. Finally, has this engagement shown a measure of consistency, or has it been patchy, uneven, slow in the beginning only to build-up in the latter portions of the programme? The best artefact to gauge this is the diligence with which a student maintained her reflection journal.
Such a picture provides an assessor with a narrative frame-of-reference of the student. It indeed makes an assessment conversation a conversation: it is very difficult, at the best of times, to assess an individual through fragments of scattered artefacts that have, most times, little connect to the particular feelings and thoughts of the person as such. This consideration is a vital one if ISDM wishes to elevate relative development (how she fared compared to herself versus others or some set standard) to a norm.
Deconstructing Reflection
A tpyical image for reflection invites us to confront seclusion, pondering, and a certain serious heavy-headedness. In case of ISDM, reflection happens in the course, or rather, midst of practice as well as outside of it. Hence, it has to be recognized that desire, deep listening and watchfulness are amongst the very first steps towards reflection: even the unprocessed experiences that a student shares at the table are very much tinged by a preliminary level of sub-conscious reflection.
It only serves to reinforce the intuition that personal mastery constitutes the bed-rock of the whole of present curriculum design. It is not an understatement that skills of self-awareness, alertness, deep listening, remembrance, writing and speaking form more than half of the student's competency as development manager. A large part of this is implied in assessment of the student's practice.
I
The remainder gets captured in what can be termed as ability to see the context for what it is: devoid of the 'I'. The process of serious heavy-headed reflection can only begin with the student's ability to recognize her instincts, impulses, prejudices, preferences, likes, dislikes, obsessions, aversion, irritation, anger, and vested desire that impinge on her thinking of the situtation.
Her ability to be aware of them and escape from them, or at least hold them at abeyance however briefly, is without doubt the spark that starts a serious reflective practice. It helps her to glimpse the situation in terms appropriate to that situtation: a doctor, to be most effective, must attend a patient in medical terms. A leader, likewise, can only look at the organisation for what it is meant or agreed to be, and not what she believes it should be.
For an assessor how best to see this in action? The above examples are sufficiently indicative: the student, in her conversations, should be able to discern the underlying nature and purpose of the situtation at hand, and keep it in mind in as consistent a manner as possible (mindfulness). There are three questions which an assessor asks herself will convey whether the student has passed this test:
1. Is her bent of mind, direction of her thinking, and conversation hint that she has internalised a perspective on development and management, so that the developmental concern, or the institutional imperative is second nature to her, or in the process of maturing within her? Or does she speak only in terms of concepts, and when pressed to the fore, gets entangled in the web of prior experiences, or personal preferences, and is unable to separate one from the other? Note: we use the term perspective and not knowledge, and the distinction is important to hold on to.
2. Is she able to accept the fact that it is ok she may not have the answers? This is a measure of her own sense of self-worth and how over-or-under-the-board she is. An assessor would rather be pleased with a candidate who knows what she does not know and why rather than one who comes up with a highly decorated and rehearsed answer.
3. Is she able to frame the situation in terms appropriate to the landscape of development management, and thereby open the possibilities for next sequence of thoughts to arise?
What can help the student develop this ability? A daily practice of personal mastery along side a degree of reading or listening in the class to acclimatise the ear and the mind to the vocabulary and perspectives of the domain.
Thus far then, our little assessment framework starts to look like this: Personal Mastery exhibited through practice combined with the ability to see the context contribute to 55% of the weightage of assessment. How to apportion it between the element of practice and the first step on reflection is debatable: a suggestion would be 35 and 25 per cent.
II
A good sub-stratum of experience to speak of, and an ability to make use of it through draining it of the 'I', allows the skill of evaluation to make way. Why evaluation? Personal mastery and perspectives will bring the student to the point where she is, at the least, able to lose hold of her baggage of the past, place herself as an objective spectator to her own context, and inject within herself a curiosity to probe that very context. Acute awareness, while to be admired in itself, is quite barren without being put to use to act - that is, to make decisions. Making decisions also means responsibility for their consequences, best noticed through the change in one's own felt context: the one which one was examining in the first place itself.
Evaluation then is an ability to frame choices appropriate to the context at hand and decide upon one to move forward. It may also be termed as a skill in critical inquiry. It is largely an intellectual endeavour and demands a few qualities: an independence of spirit (the balls to question if one may be permitted to use loose language); an endowment of common-sense; hold of a basic conceptual starting point; and some working knowledge of disciplines relevant to the context. All equal, the quality of an evaluation is a measure of student's ability to analyse, see logical inter-connections, and then knot them together as a sequence of cause and effect.
By all accounts it is an important cognitive skill, but a higher-order one. It is doubtful, whether inculcating it in any meaningful way is either possible, or really the mandate of a short-duration programme aimed at a fairly young crop. Why then give it much purchase? To contrast it with that another trait so common in our academic life: our temptation to fall for highly articulate and intelligent-sounding speech and equating it to possession of cognitive felicity.
A high IQ and high grasping power is not an automatic invitation to think critically. The assessor, more than anyone else, is actually most susceptible to the charms of the former. Critical thinking, on the other hand, is very hard to discern as the only barometer for it is the quality of conversation and its consistency: the higher the quality, the more you are assured of a serious application of mind beforehand. It requires the student to chew his thoughts, opinions, choices, and cause-effect sequence backwards and forwards more than once.
By quality we really mean that thinking has to have a flavour of authenticity and not be a sophisticated rehash of what is learnt in classroom and deftly mapped onto the present context. Questions to aid an assessor could be: if the student is asked a very pointed question related to the subject-matter of the context, what is the quality of her inference? Has she brought in any suppositions, presumptions, assumptions, or facts not related to the case at hand? If she has, can she justify their presence? In short, how true (near) is she able to remain to the context, at the same time, how far can she stray (generalise)? Thus, somewhat mercenary cross-questioning is the only tool known to man-kind for the assessor to smell this skill in the assessee.
Once again, from a stand-point of weightage: a maximum of 25% seems more than fair. Any more will raise its spectre beyond what can be strictly accomodated in the design, or expected of 25 to 30 year olds with a spartan habits of reading and writing.
III
If evaluation (critical inquiry) as a dimension of assessment presents its challenges, then the next two parameters are bound to cause commotion. For, they not only require a skilled assessor, but also a sensitive and perceptive student on the other side. One is nonetheless tempted to slot them in because they really define the bar for a decision-maker, or any half-thoughtful individual for that matter. It also helps signal the true learner from the crowd.
One of these dimensions had already found its place within the wide embrace of the previous rubric: through the trope of integration. To this, we add granularity; or, if one prefers its more accurate cousin, finesse. It is quite possible that there are students who may have shown authenticity in their practice, and a high ability for critical inquiry but may yet suffer from poverty of these two. The converse holds too: a student may not come out flying on evaluation, but may shine through dint of her perceptivity.
Granularity is a pre-condition to what today is crudely labelled innovation: a natural willingness to take an interest in finer details of things, mental persistence & concentration to extract that last remaining bit of meaning out of them; and in the process be irresitibly led to something unexpected. It is a silent quality most times; and hence, requires a trained observer to appreciate its presence in another. It is also a quality which, if put to the test, will be found to distinguish talent in very many fields.
One is not aware of techniques to help students develop this apart from building powers of concentration of the mind: meditation. Most likely, it is something the student will bring as part of her genetic endowment into ISDM. Some of the signs of its presence are: those with an interest in, or practice of, serious forms of art; those with educational pedigrees in fields that demand significant concentrated effort (theoretical mathematics & physics); or those called out by Nature as sensitive individuals in their own light.
An ISDM assessor can only test for its presence without much surity. It is also a presence that can be assimilated and felt once the student has left the room through a self-examination by the assessor: how discerning is the student? Was she able to minutely separate some of the seams of her experience, not so obvious at first? Doing so, is she able to reconstruct her context in a different light, thereby arrive at a very different reading of it than earlier, possibly a very original and unorthodox one at that? Answers to these may give the assessor some confidence to tick the box on Granularity.
IV
The final (dimension) is of course integration, though use of 'of course' is hardly natural and requires explanation. The ability to step out and take stock of the whole is in direct proportion to two factors, best illustrated through two metaphors: that of the blind elephant (exploring as many sides of the issue as possible); and the practice of absorption (or dhyan in Sanskrit) in tandem. The treachery is in the word 'tandem'.
The greater the immersion in reflective practice, the greater the likelihood of integration. On grounds of compassion one can aver: granularity is a matter of wiring of the mind; integration that of the will: grit, persistence, discipline, a desire that refuses low-hanging fruits.
What does integration really mean in context of this curriculum? The hazardous guess would be: practice fused with thought so as to become indistinguishable. Is one's manner of speaking, behaving, thinking and acting not apart from the perspectives on development and management so acquired? Have these perspectives really superseded the 'I' that the student brought into the institute without them becoming an ideological dead-weight? In short, has one become what one really thinks (himself to be)?
For it to happen, synthesis, a much in fashion word, is necessary but not sufficient. Integration is a cause of all of the above (practice, grasp of the context, evaluation, granularity) and in turn furthers their development. The analogy of a spiral is apt.
For the assesor, like in the case of granularity, the enlightenment will strike post-facto. Enlightenment is really the apt word as there are no standard questions unlike in the earlier cases: one just has to rely on one's feeling and sense that the student who just walked out could qualify for the larval stage of development management. Consensus has never covered itself with grace in history: but moments like these resurrect it. To tick off on integration, all that is needed is a yay from each member of the assessment group.
It seems loosening of standards for integration does not end here: unlike other qualities it is advisable not to grade this aspect. The strengthening of integration is, more than any other, a matter of one's starting position. Any visible progress from this starting point is a success. Though miserliness is a virtue when grading this one: if out of a batch of 60, 25 happen to tick the box on this one, then either the process of student selection has defied the statistics of the youth population, or the assessors have a bit of reflective practice of their own to do. Five per cent from a batch is where a line needs to be drawn.
Summary and a point on grading
There are thus five (or if one wishes to break practice in two, six) dimensions of assessment with their respective weightages:
1. Pratice - 35%
2. Tuning into the context - 20%
3. Evaluation (critical inquiry) - 25%
4. Granularity - 10%
5. Integration - 10%
This time a numerical scale was used to grade dimensions. It is worth considering use of descriptive stages instead, suitable for each dimension. This concurs well with the fact that the entire process of assessment is a shining example of how to apply heuristic reasoning. A precise number does much disservice to both the concept and its aesthetics.
If one is able to describe stages of development, they also serve as mnemonic devices: forcing the assessor to ask questions if she forgets; and to restrain her from getting taken in by the student's stage performance however brilliant and applause-deserving it maybe. For one should not forget: an assessment conversation has many performative elements within it. Ability to both bear this in mind and design for it ensures ISDM ticks box #2 on its own assessment framework: in other words, it does not take its own assessment process too seriously.
| |
|