సివంద మణ్ (తమిళం)

శివాజీ సిరీస్ లో భాగంగా నాకు పరిచయమైన సినిమా ఇది. ఆ కాలంలో విదేశీ షూటింగులు, వివిధ ఐరోపా దేశాల్లో తీసినందువల్ల బాగా ప్రసిద్ధి పొందిన చిత్రమని చదివాక కుతూహలం కలిగింది. అందులోనూ – హీరో హీరోయిన్లు నాకు ఇష్టమైన శివాజీ-కాంచన! వీళ్ళ జంట ఎలా ఉంటుందో? అని ఊహించుకోలేదు కాని, సినిమా థ్రిల్లర్ అని… చూడ్డం మొదలుపెట్టా.

క్లుప్తంగా కథ ఏమిటంటే – వసంతపురి అని ఒక రాజ్యం. కథాకాలం ఏమిటో నాకు అర్థం కాలేదు కానీ, ఏదో, ఆధునిక కాలమే. అందరూ ప్యాంట్లు, చొక్కాలు, మాడర్న్ దుస్తులూ వేసుకునే ఉన్నారు. మొత్తానికి పోర్టుగీసు ప్రలోభాలకి లొంగి ఈ వసంతపురి దివాను (నంబియార్) ఆ రాజ్యపు రాజుని (జావర్ సీతారామన్) కట్టడి చేసి, తానే రాజులా ప్రవర్తిస్తూ ఉంటాడు. రాజు కూతురు చిత్రలేఖ (కాంచన) స్విట్జర్లాండు లోని Zurich నగరంలో నివసిస్తూ ఉంటుంది. ఆ దేశంలోని Bern విశ్వవిద్యాలయం పరీక్షల్లో ప్రథముడిగా వచ్చాడని భారత్ (శివాజీ గణేశన్) ఫొటో పేపర్లలో పడితే చూస్తుంది. అతగాడిదీ తమ ఊరే అని, తమ రాజ్యపు inspector general కొడుకనీ తెలిసి, వెళ్ళి పరిచయం చేస్కుంటుంది. తనను తాను ఆ రాజ్యపు యువరాణిగా కాక, “వసంతి” అన్న అమ్మాయిగా పరిచయం చేసుకుంటుంది. అతగాడు రెండో/మూడో పరిచయంలోనే I love you అనేస్తాడు…వీళ్ళు గబుక్కున ప్రేమలో పడి యూరోప్ చుడుతూ ఉంటారు… మనకి మంచి లొకేషన్లలో కనిపిస్తూ ఉంటారు.

ఇదిలా ఉండగా, వసంతపురిలో యువ విప్లవకారుడు ఆనంద్ (ఆర్.ముత్తురామన్) వగైరా వ్యక్తులు దివాన్ కు వ్యతిరేకంగా పోరాడుతూ ఉంటారు. ఒకానొక సమయంలో ఆనందు మరణిస్తాడు ఈ గొడవల్లో. ఈ టైములో భారత్ ఇండియా వచ్చి ఈ విప్లవకారులతో కలిసి పోరాడాలనుకుంటాడు. తన రాజ్యం పరిస్థితి తెలియని చిత్రలేఖ భారత్ నుండి ఇదంతా తెలుసుకుని … నేనూ నీతో వస్తాను…యువరాణి నా ఫ్రెండు కనుక, తన ప్రైవేట్ ఫ్లైటులో వెళ్తే త్వరగా వెళ్ళిపోవచ్చని అతన్ని ఒప్పిస్తుంది. ఆ సందర్భంలోనే ఈవిడే యువరాణి అన్న విషయం అతనికి తెలుస్తుంది. దోవలో ఫ్లైట్లో జరిగిన ఫైట్ సీక్వెన్స్ వల్ల అది కూలిపోతుంది. వీళ్ళిద్దరు చనిపోయారని తేలుస్తారు అందరూ.

హీరో-హీరోయిన్ మధ్యలోనే చనిపోతే ఇంక కథేమిటోయ్, పుల్లాయ్! అని మనం ఎవరం అడగం. ఎందుకంటే వాళ్ళు ఎక్కడో బ్రతికే ఉంటారని మన సినీ ప్రేక్షక అనుభవం వల్ల మనకి తెలుసు కనుక! అలా బ్రతికి, తిరిగి వచ్చి, రహస్య పోరాటం చేసి, తమ రాజ్యాన్ని ఎలా కాపాడుకున్నారు? అన్నది కథ అనమాట!

(ఏడిసావులే, ఇంత కథ నువ్వే చెప్పేస్తే మేము చూట్టం దేనికి? అనుకోవచ్చు మీరు. అసలు కథంతా వాళ్ళు దేశం చేరుకుని ఆ దివాన్ని ఎదుర్కునేటప్పుడే ఉంటుంది మరి. అది ఎలా అన్నది చెప్పానా? లేదే!)

షరా మామూలుగా శివాజీ కేక. నిజానికి సినిమా చూశాక..ఇది ఎం.జీ.ఆర్. టైపు కథలా ఉందిగా…అనిపించింది. తరువాత ఎం.జీ.ఆర్. కోసమే రాసారని తెలిసింది కానీ… శివాజీ అయినా కూడా భలే చేశాడు, ఎప్పట్లాగే. మనిషి కాస్త యువకుడిగా కనబడ్డానికి చాలా కృషి చేశాడు అనుకుంటాను. సినిమా ఆద్యంతమూ body language లో చాలా తేడా ఉంది నేను చూసిన తక్కిన సినిమాలతో పోలిస్తే. పైగా, ముఖ్యంగా పాటల్లో అయితే, చాలా young and energetic గా అనిపించాడు. కాస్ట్యూంస్ కూడా తగినట్లే ఉన్నాయి. కాంచన కూడా ఆ పాత్రకి బాగా సరిపోయింది. నేను ఫ్యాన్ కనుక నాకు ఎలాగో నచ్చుతారు ఇద్దరూ, అన్నది వేరే సంగతి! పొడుగు పొడుగు డైలాగులు లేవు కానీ, శివాజీ full energy తో చేశాడు.

శివాజీ కాకుండా నన్ను ఆకట్టుకున్న నటులు – అతని తల్లి పాత్ర ధరించిన శాంతకుమారి, మెయిన్ విలన్ – నంబియార్. శాంతకుమారి కనబడేది కాసేపే అయినా కూడా, గొప్ప ప్రభావవంతంగా అనిపించింది ఆవిడ screen presence నాకు. అలాగే, శివాజీ తండ్రిగా చేసిన ఎస్.వి.ఆర్. కూడా! జావర్ సీతారామన్ నుంచి నేను ఏదో ఊహించా అందనాళ్ సినిమా చూశాక… కానీ, పాపం ఇక్కడ అతనికి అంత స్కోప్ లేదు…నిస్సహాయుడైన రాజుగా ప్రతి సీన్లోనూ దీనంగా, కోపంగా ఉండడం తప్ప.

పాటలు నాకేం అర్థం కాలేదు కానీ, ఊరికే వింటానికి, చూట్టానికి మాత్రం బాగున్నాయి. యూట్యూబులో sivandha mann songs అని వెదికితే అన్నింటికీ విడియోలు కూడా దొరుకుతాయి.

మేకప్పులు కొంచెం వెరైటీ గా ఉన్నాయి సినిమాలో. శివాజీ రకరకాల గెటప్ లలో భలే ఉన్నాడు. ముఖ్యంగా ఆ అరబ్బు అతని మేకప్ లో అయితే కేకలే!

ఇతర పాత్రల గడ్డాలు, విగ్గులు (ముఖ్యంగా ముత్తురామన్) మరీ అతికించినట్లు ఉన్నాయి. కొన్ని చిత్ర విచిత్రమైన దృశ్యాలు ఉన్నాయి. అలా ఎలా ఆలోచిస్తారా? .. అసలు ముందు వచ్చిన దానికి, దీనికి ఏమిటి సంబంధం? అనిపించినవి కొన్ని అయితే, అసలు ఈ దృశ్యం ఇంత వెరైటీగా ఎలా conceptualize చేశారు? ఇలాంటి ప్రశ్నలు చాలాసార్లే ఉదయించాయి నాకు.

ఏదేమైనా, మొత్తానికి, ఒకసారికి భేషుగ్గా చూడవచ్చు. మీరు అభిమానులైతే, ఆట్టే perfectionists కాకపోతే, రెండు మూడు నాలుగైదారేడు సార్లు చూసుకున్నా ఆశ్చర్యం లేదు! సినిమా సబ్టైటిల్స్ లేకుండా, తమిళంలో యూట్యూబులో దొరుకుతుంది. Sivantha Mann అని వెదికితే. వికీపేజీ ఇదిగో.

లిటిల్ జాన్ (అను తమిళ చిత్రం)

నాకేమిటో పి.బి.ఎస్. ని తల్చుకుంటే సింగీతం శ్రీనివాసరావు గుర్తువస్తారు. మొన్నే ఈమాటలో పి.బి.ఎస్. వ్యాసాలు చదువుతూ ఉండగా, ఇలాగే సింగీతాన్ని తల్చుకున్నాను. అప్పుడు గుర్తు వచ్చిందీ సినిమా. ఇది రిలీజైనప్పుడు మా అమ్మమ్మ వాళ్ళు అనుకుంటా, చూసి రికమెండ్ చేశారు. కానీ, అప్పట్నుంచి ఏదో టీవీలో ఒకటీ అరా సీన్లు చూడ్డమే కానీ, ఇప్పటి దాకా చూడలేదు. మొత్తానికి నిన్న చూశా.

కథ: జాన్ మెకంజీ అనే అమెరికన్ తన స్నేహితుడి కుటుంబంతో కొన్నాళ్ళు ఉండేందుకు ఇండియా వస్తాడు. అతని ముఖ్య ఉద్దేశ్యం వాళ్ళకి దగ్గర్లోని ఒక ఆలయం గురించిన పరిశోధనలు. ఆ ఆలయానికి ఒక కథ – అక్కడి అమ్మవారి విగ్రహానికి ఉన్న ముక్కెర కి ఉన్న శక్తుల మూలాన కాలభైరవుడనే వాడు వెయ్యి సంవత్సరాలుగా దాన్ని చేజిక్కించుకోవడం కోసం కాచుకుని ఉంటాడు. ఆ ఆలయంలోని పూజారి వద్ద ఉన్న శక్తుల సాయంతో ఆ ముక్కెరని కాపాడుతూ ఉంటాడు. జాన్ వచ్చాక, కాలభైరవుడి మూలాన ఆ ముక్కెర కొట్టేయబడగా, ఆ నేరం జాన్ మీదకి వెళ్తుంది. అయితే, దేవీ మహిమ వల్ల ముక్కెర కూడా ఆలయం సరిహద్దు దాటే లోపే తనను తాను రక్షించుకుని ఒక పాముల పుట్టలో పడిపోతుంది. పోలీసులు జాన్ ని అరెస్టు చేస్తారు. జాన్ తప్పించుకుని స్వామీజీ శరణు వేడితే, ఆయన జాన్ ని invisible man చేయబోయి, lilliput చేసేస్తాడు. తక్కిన కథ – జాన్ స్వామీజీ సాయంతో కాలభైరవుడిని అంతమొందించి ముక్కెర మళ్ళీ దేవి విగ్రహం వద్దకు చేర్చి, తన సాధారణరూపం రప్పించుకుని, కోరిన అమ్మాయిని పెళ్ళి చేసుకోవడం ఎలా జరిగింది? అని.

నిజానికి, ఇలాంటి కథాంశాలు బోలెడు… మన సినిమాల్లో. కానీ, ఈ సినిమాలో నాకు ప్రత్యేకం నచ్చిన అంశాలు – సంభాషణల్లోని సహజత్వం, హాస్య చతురత (జాన్ తమిళ్ మాట్లాడ్డం అసజంగానే అనిపించినప్పటికీ!). అలాగే, అనుపం ఖేర్ ను స్వామీజీ గా ఎందుకు తీసుకోవాలనుకున్నారో గానీ, విలక్షణంగా భలే ఉన్నాడు ఆ గెటప్ లో. నాజర్-ఆర్.ఎస్.శివాజీ లు కనబడ్డంతసేపు బాగా కాలక్షేపమైంది. జాన్ లిల్లిపుట్ అయే సీన్ బాగా తీసినట్లు తోచింది. కాలభైరవుడిగా ప్రకాశ్ రాజ్ కాసేపు భయపెట్టాడు కానీ, మెయిన్ విలన్ కి ఉండాల్సినంత పాత్రలేదు అసలు కాలభైరవుడికి ఈ కథలో. కానీ, సినిమా కథ నాకు మరీ సింపుల్ గా అనిపించింది. క్లైమాక్సు ఫైట్ కూడా మరీ తేలిగ్గా ముగిసిపోయినట్లు అనిపించింది. ఇంతకీ సినిమాలో హీరోయిన్ జ్యోతిక. ఆట్టే రోల్ లేదు కానీ, ఉన్నంతలో బాగానే ఉంది.

నాకాట్టే వివరంగా చర్చించే ఓపిక లేదు కానీ, మొత్తానికైతే సినిమా కాలక్షేపానికి చూడ్డానికి బాగుంది. ఆ పాటలు పెట్టకుండా ఉండి ఉంటే ఇంకా బాగుండేది (పాటలు బానే ఉన్నాయి వినడానికి. కానీ, సినిమాలో రావడమే బాలేదు). సబ్టైటిల్స్ లేకుండా, తొమ్మిది భాగాల్లో యూట్యూబులో దొరుకుతుంది సినిమా.

పి.బి.శ్రీనివాస్ ఉదయం – ఈమాట.కాం వల్ల!

నిన్నటి ఉదయం అంతా ఈమాట పత్రికలో పి.బి.శ్రీనివాస్ గారిపై వచ్చిన వ్యాసాలు చదువుతూ, వారి పాటలు వింటూ గడిచింది. అక్కడికి నేను ఏదో ఆయన వీరాభిమానిని అనుకునేరు. ఏదో, అందరి పాటలు వింటూనే ఆయనవీ వినడమే గానీ, ఆయన రంగు రంగుల పెన్నులు పెట్టుకుంటారు, జరీ తలపాగా ధరిస్తారు, పలు భాషల్లో నిష్ణాతుడు, వాళ్ళ మనవడు బాలనటుడిగా చేశాడు అన్నవి తప్పిస్తే నాకు ఆయన గురించి ఏమీ తెలియదు. ఈమాట.కాం వెబ్ పత్రిక వారు ఈ నెల సంచికలో పి.బి.ఎస్. గురించి ప్రత్యేక వ్యాసాలు, పి.బి.ఎస్. స్వయంగా రాసిన వ్యాసాలు చేర్చడం తో, నేను ఆసక్తిగా చదవడం మొదలుపెట్టాను.

ఇవిగో ఆ వ్యాసాలు:

1. ప్లే బ్యాక్ సింగర్ పి.బి.ఎస్ అని – విష్ణుభొట్ల లక్ష్మన్న, జెజ్జాల కృష్ణమోహన రావు, పరుచూరి శ్రీనివాస్ గార్లు రాసిన వ్యాసం : ఇందులో ఒక యాభై పాటలకన్నా లంకెలు ఉండి ఉంటాయి … నాలుగైదు భాషల నుండి. చాలామట్టుకు వాటి గురించిన వివరాలు కూడా జతచేశారు. చాలా తక్కువ వ్యవధిలో (కళ్ళుతిరిగేలా) బోలెడు పాటల వివరాలతో కూర్చిన వ్యాసం ఇది. ఈ వ్యాసం చదువుతూ, ఆ పాటలు వింటూ పూర్తి చేయాలంటే ఒక నాలుగైదు గంటలు పట్టొచ్చు కనుక, వ్యాసం చదువబోయేవారు దానికి సిద్ధం కండి.

2. సంగీత సాహిత్య శ్రీనివాసుడు - ఏల్చూరి మురళీధరరావు గారి వ్యాసం: చదవగానే… “వావ్!” అనుకున్నా. పి.బి.ఎస్. లోని మేధావి ని పరిచయం చేసిన వ్యాసం… నాకు మొత్తం అర్థమైపోయిందని చెప్పలేను కానీ, ఇది మాత్రం అద్భుతమైన వ్యాసం. బహుశా, నేను మళ్ళీ వచ్చి మళ్ళీ మళ్ళీ చదవబోయే వ్యాసం అనుకుంటున్నాను.

3. హాయిగా పాడుదునా – సాలూరి రాజేశ్వర రావు పై పి.బి.ఎస్. విజయచిత్రలో 60లలో రాసిన వ్యాసం. ఆసక్తికరంగా ఉంది. పి.బి.ఎస్. ఆట్టే క్లిష్టత లేకుండా, మానవభాషలో రాశారు కనుక నా బోంట్లకి కూడా అర్థమైపోయిందని సంతోషం!

4. చలనచిత్ర సంగీత సాహిత్య పోకడలు – మరో పి.బి.ఎస్. వ్యాసం. ఇది కూడా ఆసక్తికరంగా చదివించింది.

- ఆ విధంగా, నా శనివారం ఉదయాన్ని పి.బి.ఎస్., ఈమాట ఆక్రమించారు. నేను ఆరింటికేమో మొదలుపెట్టి, ఈ కార్యక్రమం ముగించుకుని తల తిప్పేసరికి 12 కావొస్తోందని గ్రహించా! ఈ మధ్య కాలంలో తెలుగు, అందునా పత్రికలు, అందునా ఈమాట జోలికి అసలు వెళ్ళలేదు కనుక, నా స్థాయికి ఇది రికార్డే అనిపించి, ఇలా బ్లాగు రాసుకుంటున్నా

Automated Grading and Counter Arguments-4 (last)

(Continued from Part-3)
***

Finding 9: machine scoring shows a bias against second-language writers (Chen & Cheng, 2008) and minority writers such as Hispanics and African Americans (Elliot, Deess, Rudniy, & Joshi., 2012]

Report-1: The best part about this report is the stance it takes. It immediately got me interested in it.

“Given the fact that many AWE programs have already been in use and involve multiple stakeholders, a blanket rejection of these products may not be a viable, practical stand.
…
..
A more pressing question, accordingly, is probably not whether AWE should be used but how this new technology can be used to achieve more desirable learning outcomes while avoiding potential harms that may result from limitations inherent in the technology.”

-This exactly has been my problem with these statements on humanreaders.org website.

The study primarily supports the idea of integrating human and machine assessments, by taking advantage of good things in both, as mentioned below:

“The AWE implementation was viewed comparatively more favorably when the program was used to facilitate students’ early drafting and revising process, and when the teacher made a policy of asking students to meet a preliminary required standard and subsequently provided human feedback. The integration of automated assessment and human assessment for formative learning offers three advantages….”

At least I did not find a direct mention about a “bias” against second language writers in this report! We need to stretch our imagination a bit to reach that conclusion!

Report-2: – The second report was already mentioned in Finding 7. Like before, I did not find a direct relevance of these results to this “finding”. However, I see the point in raising this issue. But, what I don’t understand is that this is just like some American coming and correcting Indian English So, this kind of “bias” can exist in humans as well. What really is a way to handle this, manually or automatically? This does not make the case favourable to human assessment (IMHO).

Finding 10: for all these reasons, machine scores predict future academic success abysmally (Mattern & Packman, 2009; Matzen & Hoyt, 2004; Ramineni & Williamson, 2013 – not freely accessible)

- I actually did not go through these references beyond their intro and conclusion sections as I felt that “finding” is too much of a blanket statement to be connected to these findings. Skimming through the two freely accessible reports among these confirmed my suspicion. These reports focus more on doing a critical analysis of automated systems and suggesting ways to improve them and combine them with some other things… and not say “machine scores predict future academic success abysmally”.

Part-2 of these findings were focused on the statement: “machine scoring does not measure, and therefore does not promote, authentic acts of writing”

While I intend to stop here (partly because its so time consuming to go through so many reports and partly because of a growing feeling of irritation with these claims), some of these findings-part 2 made me think and some made me smile and some made me fire a question back…

Those that made me think:
“students who know that they are writing only for a machine may be tempted to turn their writing into a game, trying to fool the machine into producing a higher score, which is easily done “
-This is something that recurs in my thoughts each time I think of automated assessment.. and I know it is not super-difficult to fool the machine in some cases.

“as a result, the machine grading of high-stakes writing assessments seriously degrades instruction in writing (Perelman, 2012a), since teachers have strong incentives to train students in the writing of long verbose prose, the memorization of lists of lengthy and rarely used words, the fabrication rather than the researching of supporting information, in short, to dumb down student writing.”
- This part is actually the main issue. Rather than focusing on just making blanket claims and pushing everything aside, humanreaders.org or any such initiatives should understand the inevitability of automated assessment and focus on how to combine it with human evaluation and other means better!

Those that rather made me smile, although I can’t brush aside these things as impossible

“students are subjected to a high-stakes response to their writing by a device that, in fact, cannot read, as even testing firms admit.”

“in machine-scored testing, often students falsely assume that their writing samples will be read by humans with a human’s insightful understanding”

“teachers are coerced into teaching the writing traits that they know the machine will count .. .. and into not teaching the major traits of successful writing.. .. “

- No I don’t have any specific comments on these. But, in parts, this is very imaginative…and in parts, it is not entirely impossible.

Those that gave new questions:
“conversely, students who knowingly write for a machine are placed in a bind since they cannot know what qualities of writing the machine will react to positively or negatively, the specific algorithms being closely guarded secrets of the testing firms (Frank, 1992; Rubin & O’Looney, 1990)—a bind made worse when their essay will be rated by both a human and a machine”
-as if we know what humans expect! I thought I wrote very a good essay on Sri Sri in my SSC exam’s question “my favourite poet”… and I got less marks (compared to my past performances) in Telugu, of all things. Apparently, teachers in school and those external examiners did not think alike! (or probably the examiner was a SriSri hater!)

“machines also cannot measure authentic audience awareness”
- Who can? Can humans do that with fellow humans? I don’t think so. I know there are people who think I am dumb. There are also those who think I am smart. There are also those who think I am mediocre. Who is right?

Conclusion of this series:

Although I did not do a real research-like reading and this is not some peer-reviewed article series, I spent some time doing this and it has been an enriching experience in terms of the insights it provided on the field of automated assessment and its criticisms.

What I learnt are two things:
* Automated Assessment is necessary and could not be avoided in future (among several reasons, because of the sheer number of students compared to the number of trained evaluators)
* Overcoming the flaws of automated assessments and efficient ways to combine it with trained human evaluators is more important, realistic and challenging than just branding everything as rubbish.

Although the above two are rather obvious, humanreaders.org “findings” and the way such “findings” immedietly grab media attention convinced me about the above two things more than before!

Tere Bin Laden – good one!

I watched “London, Paris, New York” the other day and got to “Tere Bin Laden” from there (with the Ali Zafar connection). Despite my poor knowledge of films in general and Bollywood movies in particular, I vaguely remember reading good reviews about this movie back then and the storyline from wiki fascinated me. So, I started watching and I should say two major reasons why I liked the movie are: the movie is short and it is a thoroughly entertaining satire.

What I really liked about the movie were the screenplay and the dialogues. Right from Ali Hassan’s intro scene to the sequence with FBI and Pakistani Intelligence folks, everything was so funny and satirical at the same time. Pradhyuman Singh as Noora/Laden and Chinmay Mandlekar as the Pakistani intelligence officer were the best among the actors (IMHO) although everyone suited their role very well.

I don’t know why but I was reminded of Art Buchwald‘s “Beating around the Bush” column essays, several of which had some satirical comments about the “war on terror”. I should perhaps read those essays again! I was also reminded of another brilliant satirical film “The President is Coming“.

Those who are interested in knowing more about the movie can read the story (with all the spoilers) in the wiki page. Those who want to know my (in)valuable commentary about the movie, well, here it is: just watch it!

(If someone is relieved that I would now stop writing those overtly research like posts which no one except me are interested in reading, you are mistaken!)

Automated Grading and Counter Arguments-3

(Continued from part-2. All parts can be seen here)
***

Finding 5: machines require artificial essays finished within very short time frames (20-45 minutes) on topics of which student writers have no prior knowledge (Bridgeman, Trapani, & Yigal, 2012; Cindy, 2007; Jones, 2006; Perelman, 2012b; Streeter, Psotka, Laham, & MacCuish, 2002; Wang, & Brown, 2008; Wohlpart, Lindsey, & Rademacher, 2008)

First, Second, Third, Fourth, 7th references were not freely accessible.

Fifth Report (just search with the title in google): The primary conclusion of this report is: “The automated grading software performed as well as the better instructors in both trials, and well enough to be usefully applied to military instruction. The lower reliabilities observed in these essay sets reflect different instructors applying different criteria in grading these assignments. The statistical models that are created to do automated grading are also limited by the variability in the human grades.”

6th Report
This report studied the correlation between the machines and human raters and concluded, contrary to all previous studies, that there is no significant correlation between them.

While this was an interesting conclusion, I had a few questions which I could not clarify better as those references couldn’t be found online (for free). Here are the questions I have.

1. I think all these Automated Scoring systems work well within their framework. Like, the GRE exam grading works typically for GRE exam like essays but not for say, scoring what I write in my blog. But, they all are really customizable to any such target-specific responses (The NYT article on EdX, which began this series of blog posts from me makes that clear. It needs 100 sample essays and it will learn how to score essays from 101th on, to put it simply.).. So, is it fair to test Intellimetric or whatever system, on some essays that are about something else…and from other domain?

2. As far as I remember, all those other reports used pearson’s correlation. This report used spearman’s correlation. Can we compare the numbers we get directly? Why cant both be included?

-so far except the 4th one, the others seemed to be more about how well the automated scores approximate human judgements (looking at all the abstracts, which are freely available to read!!)

Finding 6: in these short trivial essays, mere length becomes a major determinant of score by both human and machine graders (Chodorow & Burstein, 2004; Perelman, 2012b)

First Report:
I actually found this report to be a very interesting read, for several reasons. Some of them are mentioned below:

“Since February 1999, ETS has used e-rater as one of the two initial readers for the GMAT writing assessments, and, in this capacity, it has scored more than 1 million essays. E-rater’s scores either match or are within one point of the human reader scores about 96% of the time”
-I thought this is an amazing performance (especially with such a high percentage even after grading 1 million essays or more!). I also like this part where they compare the machine-human agreement with human-human agreement, which is why I again think that no single entity should be the sole-scorer (there should be either 2 humans, 2 differently built machines or machine-human combinations).

While I am not completely comfortable with an automated system being the sole scorer in the low-stakes high school exams etc, I have the same feeling about human evaluators too. I think bias can exist both in a human and in a machine (in a machine, because its intelligence is limited. It learns only what it sees). I guess at that level, you always have an option to go back and demand a re-evaluation?

Primary conclusion of this report was that: “In practical terms, e-rater01 differs from human readers by only a very small amount in exact agreement, and it is indistinguishable from human readers in adjacent agreement. But despite these similarities, human readers and e-rater are not the same. When length is removed, human readers share more variance than e-rater01 shares with HR.”
-I think this is what the humanreaders.org use to come up with this finding. I do not know how to interpret this particular finding although it disappoints me a bit. Yet, the ETS report made a very interesting read.

A second conclusion that this “finding” does not focus on, but I found interesting, was the effect of the test taker’s Native Language on the overall scores. “There is a main effect for language in the data of the current study. The mixed and Spanish groups have higher scores on average than the Arabic and Japanese. These
differences remain even when length is removed. For most of the prompts, e-rater01 shows the same pattern of differences across native language groups as HR, even with length differences partialed out. Future work should include additional language groups to see if these results generalize.”
- After the recent Native Language Identification shared task, I got really curious on this very topic – the interaction between the native language and the scores that the test takers get. I think this is something that I might need to study further, irrespective of the disappointments during these readings!

Again, only the first report (from ETS) is freely accessible. If the purpose of humanreaders.org was to actually convey their message to a “curious” commoner, I think they are repeatedly failing in this by providing such inaccessible references. ….

Finding 7: machines are not able to approximate human scores for essays that do fit real-world writing conditions; instead, machines fail badly in rating essays written in these situations (Bridgeman, Trapani, & Yigal, 2012; Cindy, 2007; Condon, 2013; Elliot, Deess, Rudniy, & Joshi, 2012; Jones, 2006; Perelman, 2012b; Powers, Burstein, Chodorow, Fowles, & Kukich, 2002; Streeter, Psotka, Laham, & MacCuish, 2002; Wang & Brown, 2008; Wohlpart, Lindsey, & Rademacher, 2008)

First report, second, third, fifth, sixth, seventh, tenth reports are not freely accessible (although looking at the abstracts, I did not get an impression that these reports are against automated scoring in any way!).

A version of 4th report is available. Even this did not give me an impression that it has anything against automated scoring as such.. although there is a critical analysis of where automated scoring fails and it ended with a call for better systems.

(8th and 9th reports are already discussed in Finding 5)

-Again, I understand that this is an issue … but I don’t understand how just avoiding automated scoring will solve it.

Finding 8: high correlations between human scores and machine scores reported by testing firms are achieved, in part, when the testing firms train the humans to read like the machine, for instance, by directing the humans to disregard the truth or accuracy of assertions (Perelman, 2012b), and by requiring both machines and humans to use scoring scales of extreme simplicity

- Like I said before, I could not find a freely accessible page for (Perelman, 2012b). It would have been great if some freely accessible report was provided as additional reading here… because this claim/finding is among those most intriguing ones. First, they say that the machines are bad. Now, they say even the humans are bad! Should we be evaluated at all or not?

Conclusion for this part:
… And so, although its getting more and more disappointing… I decided to continue with these reports further because, even if the analysis and those claims (of humanreaders.org) are flawed according to me, I find those “findings” (or whatever they are) worth pondering about. There may not be a real solution. But, some of these issues need some thinking on the part of people who are capable of doing it better (e.g., teachers and researchers in education, researchers in automated scoring).

(to be continued..)

Automated grading and counter arguments-2

(Continued from part-1)
***

So, I started to read their research findings.
(It would have been nice if they also gave online links to those references. Nevertheless, it was not very difficult to google most of them).

Finding 1: computer algorithms cannot recognize the most important qualities of good writing, such as truthfulness, tone, complex organization, logical thinking, or ideas new and germane to the topic (Byrne, Tang, Truduc, & Tang, 2010)

(This report referred here can be read here)
In this, after a brief discussion on the development of eGrader, an automatic grading system, they conclude with the following words, after deciding to not use it for automatic grading.
“The machine reader appears to penalize those students we want to nurture, those who think and write in original or different ways. For us, the subjective element which was as important as the objective aspects of the essays, proved too complex to measure.”

-It is indeed a matter of concern (this above statement) …. but, I wondered if there really was sufficient evidence (at least in this referred report) to extrapolate this observation with a small sample of 33 students to phrase the Finding 1.

10 students in this report came back to the human evaluators for a regrading and got better grades after that. But, won’t this sort of disagreement also happen if there are human evaluators instead of machines? Can we be sure that two humans always give the same score to a document without disagreement? What will we do when there is a disagreement between two human scores? May be we call a third evaluator? Isn’t that what they do with machines too now?

Also, I guess automated scoring, wherever its used in a high stakes assessment scenario (like…grading in an entrance exam or in some competitive proficiency test etc), is not used as a sole decider. It seems to be coupled with atleast one human judge (both GRE and GMAT automatic assessment has atleast one human judge and a second one will be called if there is a disagreement between human and machine scores).

So, what I understood: the “finding 1″ might still be true … but this document that is referred there is not related to that statement. Its more like a “null hypothesis” than a “finding”.

Finding 2. to measure important writing skills, machines use algorithms that are so reductive as to be absurd: sophistication of vocabulary is reduced to the average length or relative infrequency of words, or development of ideas is reduced to average sentences per paragraph (Perelman, 2012b; Quinlan, Higgins, & Wolff, 2009)

(I could not trace a freely accessible link to the first reference. Second one here).

Its amazing that the same system that uses so many different measures to estimate various aspects of the language quality of an essay (see the last 4 pages of the pdf above) uses a relatively “surface” measure of lexical sophistication and development (No, I don’t have any better ways to measure it. Don’t ask me such questions!!). However, I think “prompt specific vocabulary usage” description in the end actually handles the sophistication of vocabulary part to some extent. And there is always atleast one human grader to identify things like “out of the box” thinking or novel word usages that are relevant and also competent. These automatic assessment systems don’t seem to be the sole decision makers anyway!

So, I don’t understand the issue, again. Further, I am amazed that so many other positive things from this second report were completely ignored and directly a “finding 2″ was finalized by skipping all those!

Finding 3 machines over-emphasize grammatical and stylistic errors (Cheville, 2004) yet miss or misidentify such errors at intolerable rates (Herrington & Moran, 2012)

(The first reference does not seem to be freely available.)
-It was clear from the beginning that the authors of the second paper have a strong bias against automated scoring. Its entirely acceptable .. everyone can have some opinion. But, then, we can’t expect that paper to be objective then, can we? Atleast I did not find it objective. I thought there would be some form of well defined scientific study there. But, all they discussed was one single student essay and generalized that to the performance of the automated system as a whole (I wonder why that is not possible to do even the other way round by finding an essay where it works and conclude equally confidently that automated system is the best! ). Further, I felt that an analysis as to why the automated system showed those errors was not performed. A major criticism about spelling was that the machine identified words like “texting”, “i.e.” as spelling errors. But, if “i.e.” was supposed to be written as “i.e.,” and texting is not a word in say, English dictionary, I think this should be expected. Infact, I would guess that a very conservative human evaluator too might point these things out. So, based on this single student essay analysis (some of which is debatable), its concluded that the tool should be banished from classrooms… (this is where I start getting disillusioned… should I really continue these readings?? They seem so biased and the analysis seems more like over-generalization than science anyway!)

The only interesting point in this report was about the machine’s bias towards standard American English. I am curious to know more on this aspect. Actually, I did find the starting premise (that the machines overemphasize stylistic errors) interesting… but this report did not live up to its own promise in terms of the quality of analysis provided.

Finding 4 machines cannot score writing tasks long and complex enough to represent levels of writing proficiency or performance acceptable in school, college, or the workplace (Bennett, 2006; Condon, 2013; McCurry, 2010; Perelman, 2012a)

Report 1 – Bennett, 2006: There are two major claims in this. 1) “Larger differences in computer familiarity between students with the same paper writing proficiency would be associated with correspondingly bigger discrepancies in computer writing scores” and 2) “The weighting of text features derived by an automated scoring system may not be the same as the one that would result from the judgments of writing expert”
-Actually, although 1) seems rather obvious (as we need to “type” essays on a computer), there is no real solution that this report proposes. 2) Ofcourse, the weighting would be different between humans and machines. Machines don’t learn like humans and humans don’t learn like machines! But, when there is not so much of discrepancy between both their scores…and so long as the users are satisfied, I guess this is something that we can live with. Anyway, the paper did not suggest any better alternative.

Condon, 2013; McCurry, 2010 are not freely accessible.

Report4 – Perelman, 2012a: The conclusion of this report is that the way automated essay scoring systems evaluate the language constructs is not actually the way actual writing teachers do it. Although this is an important point to be addressed,
the other person in me is always ready to say: what we learn and what the machine learn need not always be the same. Our routes to reach the same conclusion might really not cross at all!

Concluding this part:
In a sense the whole thing is really amazing. On one hand, they talk about the shortage of human evaluators to grade student scripts. On the other hand, they want a ban on automated assessment. I wonder what exactly is the point of humanreaders.org, even after reading so many reports! I don’t understand their solution, if there is any, yet.

The other claims and their associated reports might help (I hope!)

(To be continued)

Automated grading and Counter arguments-1

Take 1: I read this NYT article about EdX’s announcement that it will release its automatic grading software “free on the web, to any institution that wants to use it”.(Article can be read here)

I particularly also liked this part of the statement:

“The EdX assessment tool requires human teachers, or graders, to first grade 100 essays or essay questions. The system then uses a variety of machine-learning techniques to train itself to be able to grade any number of essays or answers automatically and almost instantaneously.”

Take 2: There is this organization called humanreaders.org. According to the news report, “The group, which calls itself Professionals Against Machine Scoring of Student Essays in High-Stakes Assessment, has collected nearly 2,000 signatures, including some from luminaries like Noam Chomsky.”

Given my recent interest on “evaluation of evaluation”, this particular statement from one of the people from this group caught my attention:

“My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman, a retired director of writing and a current researcher at M.I.T.”

Take 3: I end up navigating through the pages of humanreaders.org and read their reports and conclusions, purely because of the above statement.

Take 4: This post comes out. … and seems to become a couple of posts soon.
****
So, the humanreaders.org’s major claim is:
“We call for schools, colleges, and educational assessment programs to stop using computer scoring of student essays written during high-stakes tests.”

As someone not doing anything directly with automated scoring but having academic interest in it owing to its proximity to what I do, I was naturally curious after seeing such a strong statement.

At this point, I have to state what I think about it. I think automated scoring is a nice complimentary system to have, along with human evaluators. This is also why I like the GRE/GMAT-AWA section style scoring model. For example, here is what they say on the ETS website, about GRE essay scoring:

“For the Analytical Writing section, each essay receives a score from at least one trained reader, using a six-point holistic scale. In holistic scoring, readers are trained to assign scores on the basis of the overall quality of an essay in response to the assigned task. The essay score is then reviewed by e-rater, a computerized program developed by ETS, which is used to monitor the human reader. If the e-rater evaluation and the human score agree, the human score is used as the final score. If they disagree by a certain amount, a second human score is obtained, and the final score is the average of the two human scores.”
(Link with more detailed explanation here)

Also, in the context of MOOCs and the sheer number of students that enrol in them, perhaps, its a worthwhile idea to explore ways of evaluating them better. Surely, when you are offering courses for free or minimal charges and you have thousands of students, you cannot afford to manually grade each and every student test script. I do like the idea of peer-reviewed essay grading too, though.

Coming back to the topic, the humanwriters.org homepage continued:

Independent and industry studies show that by its nature computerized essay rating is

* trivial, rating essays only on surface features such as word size, topic vocabulary, and essay length
* reductive, handling extended prose written only at a grade-school level
* inaccurate, missing much error in student writing and finding much error where it does not exist
* undiagnostic, correlating hardly at all with subsequent writing performance
* unfair, discriminating against minority groups and second-language writers
* secretive, with testing companies blocking independent research into their products

-It is here, that I began feeling … “not true… not true…something is missing”… reasons? While for some of them, I would need a more detailed reading, I was surprised to see some of the other points above:

trivial: I did spend some time in the past few months, reading published (and peer-reviewed) research on this kind of systems (e-rater, for example) and at least I feel that its not really “trivial”. We can always argue -
a) “this is not how a human mind does it”
b) “there is much more than what you do now”.
(I am reminded of the Norvig-Chomsky debate as I write these above two lines!)
But, IMHO, we still cannot call the current state-of-the-art “trivial”. If it is so trivial, why is it that so many researchers still spend major part of their working hours on handling this problem?

unfair: Even if I start believing that its true, I don’t understand how we can be so sure that a human evaluator too won’t do this?

secretive: On this part, I partly agree. But, these days, there are so many competitions on automated assessment (eg: 2012 automated essay scoring competition by the Hewlett Foundation, the Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge at SemEval 2013, Question Answering for Machine Reading Evaluation at CLEF 2013) and people from the industry also participate in these competitions, as far as I noticed. So, although one might still not be able to see the actual working of the respective company products or their actual student texts (hello, they are companies and like many other companies, they too have some proprietary stuff!), these competitions actually provide a scope for open research, fair play and also a scope to explore various dimensions of automated essay scoring. After just browsing through some of these things, I really can’t call things trivial… secretive, they may be… stupid, they certainly are not!

So… I ended up reading their “research findings” as well. As I started reading some of the references, I understood the power of selective reporting, again! By selectively reporting what we choose to report, we can always turn everything in our favor… and this realization perhaps is making me write these posts.

(continued)

PS 1: What qualification do you have? someone might ask me. I have the necessary background to understand research documents on this topic. I did some hobby experiments with this sort of stuff, with a freely accessible exam dataset and have an idea of what works and why it works when it works. I never worked on any real automated scoring system. I have no more interest than this on this topic, at least as of now.

Oh My God! (oh, its a movie!)

So I saw “Oh My God!” as usual, a few months late. Whenever I think of this movie, I am reminded of another well-made entertainer with the same name (and without the exclamation) with Vinay Pathak in the lead role. Although I do not understand the fun in having the same name for movies that came in a rather short span (and also, not having a more desi sounding name), I won’t get in to that now.

Some people say that this is based on a Gujarathi play (Kanji Virudh Kanji) which is based on another Hindi play. Some people say its a free-make of “The Man who sued God“. Whatever it is, its a good watch and very entertaining too.

Dialogues are very well written. If I watch the movie again at any time, dialogues are a major reason. Paresh Rawal was as usual brilliant but actors like Govind Namdeo were a bit over the top as time passed… but who really amazed me was … Mithun Chakraborthy! He was brilliant as this evil God man with rather weird mannerisms. This is really why I write this post. To say that I am amazed by Mithun’s expressions!

End to story!

Machine Learning that Matters – Some thoughts.

Its almost an year since Praneeth sent this paper and I read it…and began blogging about it. I began re-reading it today, as a part of my “evaluating the evaluation” readings, and thought I still have something to say (largely to myself) on some of the points mentioned in this paper.

Machine Learning that Matters
by Kiri L. Wagstaff
Published in proceedings of ICML 2012.

This is how it begins:

“Much of current machine learning (ML) research has lost its connection to problems of import to the larger world of science and society”
-I guess the tone and intention of this paper is pretty clear in this first sentence.

I don’t have any issues with the tone as such – but I thought there are so many real-world applications of machine learning these days! That doesn’t mean that every machine learning research problem leads to solving a real-world problem though, which holds good for any research. So, the above statement in my view can apply to any research in general.

I was fascinated by this statistics on the hyper-focus on bench marked datasets.

A survey of the 152 non-cross-conference papers published at ICML 2011 reveals:
148/152 (93%) include experiments of some sort
57/148 (39%) use synthetic data
55/148 (37%) use UCI data
34/148 (23%) use ONLY UCI and/or synthetic data
1/148 (1%) interpret results in domain context

-Since I am not into machine learning research but only use ML for computational linguistics problems, I found this to be very interesting… and a very valid point.

Then, the discussion moves on to evaluation metrics:

“These metrics are abstract in that they explicitly ignore or remove problem-specific details, usually so that numbers can be compared across domains. Does this seemingly obvious strategy provide us with useful information?”

-In the discussion that followed, there were some interesting points on what various evaluation metrics fail to capture etc. I have been reading on this topic of evaluation metrics for supervised machine learning in the recent past…and like with those, I am left with the same question even here – what is the best evaluation, then? Ofcourse, “real world”. But, how do you quantify that? How can there be some kind of evaluation metric, thats truly comparable with other peer research groups?

I got my answer in the later part of the paper:

Yet (as noted earlier) the common approach of using the same metric for all domains relies on an unstated, and usually unfounded, assumption that it is possible to equate an x% improvement in one domain with that in another. Instead, if the same method can yield profit improvements of $10,000 per year for an auto-tire business as well as the avoidance of 300 unnecessary surgical interventions per year, then it will have demonstrated a powerful, wide-ranging utility.

Next part of the discussion is on identifying where machine learning matters:

“It is very hard to identify a problem for which machine learning may offer a solution, determine what data should be collected, select or extract relevant features, choose an appropriate learning method, select an evaluation method, interpret the results, involve domain experts, publicize the results to the relevant scientific community, persuade users to adopt the technique, and (only then) to truly have made a difference”

-Now, I like that.

I also liked this point on the involvement of the world outside ML.

“We could also solicit short “Comment” papers, to accompany the publication of a new ML advance, that are authored by researchers with relevant domain expertise but who were uninvolved with the ML research. They could provide an independent assessment of the performance, utility, and impact of the work. As an additional benefit, this informs new communities about how, and how well, ML methods work.”

“Finally, we should consider potential impact when selecting which research problems to tackle, not merely how interesting or challenging they are from the ML perspective. How many people, species, countries, or square meters would be impacted by a solution to the problem? What level of performance would constitute a meaningful improvement over the status quo?”

-Well, I personally share the sentiments expressed here. I like and I want to work on problems whose solutions can possibly have a real life impact. However, I consider it my personal choice. But, I don’t understand what is wrong in doing something because its challenging! What’s wrong in researching for fact finding? There will be practical implications to certain research problems. There might not be an immediate impact for some. There might not be a direct impact for some. There might not really be a practical impact for some. But should that be the only deciding factor? (Well, of course, when the researchers are funded from public taxes, perhaps its expected to be thus. But, should it be thus, always??)

I found the six old and new Machine learning impact challenges really interesting.
Here are the new ones from the paper:

1. A law passed or legal decision made that relies on the result of an ML analysis.
2. $100M saved through improved decision making provided by an ML system.
3. A conflict between nations averted through high-quality translation provided by an ML system.
4. A 50% reduction in cybersecurity break-ins through ML defenses.
5. A human life saved through a diagnosis or intervention recommended by an ML system.
6. Improvement of 10% in one country’s Human Development Index (HDI) (Anand & Sen,1994) attributable to an ML system.

And finally, I found the last discussion on obstacles to ML impact also to be very true. I don’t know why there is so little work making machine learning output comprehensible to its users (e.g., doctors using a classifier to identify certain traits in a patient might not really want to see an SVM output and take a decision without understanding the output!) (atleast, I did not find too much work on Human Comprehensible Machine Learning)

As I read it again and again, this paper seems to me like a Theory vs Practice debate (generally speaking) and can possibly be worth reading for anyone outside machine learning community too (like it was useful for me!).

******
End disclaimer: All those thoughts expressed are my individual feelings and are not related to my employer.:-)