(Continued from Part-3)
***

Finding 9: machine scoring shows a bias against second-language writers (Chen & Cheng, 2008) and minority writers such as Hispanics and African Americans (Elliot, Deess, Rudniy, & Joshi., 2012]

Report-1: The best part about this report is the stance it takes. It immediately got me interested in it.

“Given the fact that many AWE programs have already been in use and involve multiple stakeholders, a blanket rejection of these products may not be a viable, practical stand.
…
..
A more pressing question, accordingly, is probably not whether AWE should be used but how this new technology can be used to achieve more desirable learning outcomes while avoiding potential harms that may result from limitations inherent in the technology.”

-This exactly has been my problem with these statements on humanreaders.org website.

The study primarily supports the idea of integrating human and machine assessments, by taking advantage of good things in both, as mentioned below:

“The AWE implementation was viewed comparatively more favorably when the program was used to facilitate students’ early drafting and revising process, and when the teacher made a policy of asking students to meet a preliminary required standard and subsequently provided human feedback. The integration of automated assessment and human assessment for formative learning offers three advantages….”

At least I did not find a direct mention about a “bias” against second language writers in this report! We need to stretch our imagination a bit to reach that conclusion!

Report-2: – The second report was already mentioned in Finding 7. Like before, I did not find a direct relevance of these results to this “finding”. However, I see the point in raising this issue. But, what I don’t understand is that this is just like some American coming and correcting Indian English So, this kind of “bias” can exist in humans as well. What really is a way to handle this, manually or automatically? This does not make the case favourable to human assessment (IMHO).

Finding 10: for all these reasons, machine scores predict future academic success abysmally (Mattern & Packman, 2009; Matzen & Hoyt, 2004; Ramineni & Williamson, 2013 – not freely accessible)

- I actually did not go through these references beyond their intro and conclusion sections as I felt that “finding” is too much of a blanket statement to be connected to these findings. Skimming through the two freely accessible reports among these confirmed my suspicion. These reports focus more on doing a critical analysis of automated systems and suggesting ways to improve them and combine them with some other things… and not say “machine scores predict future academic success abysmally”.

Part-2 of these findings were focused on the statement: “machine scoring does not measure, and therefore does not promote, authentic acts of writing”

While I intend to stop here (partly because its so time consuming to go through so many reports and partly because of a growing feeling of irritation with these claims), some of these findings-part 2 made me think and some made me smile and some made me fire a question back…

Those that made me think:
“students who know that they are writing only for a machine may be tempted to turn their writing into a game, trying to fool the machine into producing a higher score, which is easily done “
-This is something that recurs in my thoughts each time I think of automated assessment.. and I know it is not super-difficult to fool the machine in some cases.

“as a result, the machine grading of high-stakes writing assessments seriously degrades instruction in writing (Perelman, 2012a), since teachers have strong incentives to train students in the writing of long verbose prose, the memorization of lists of lengthy and rarely used words, the fabrication rather than the researching of supporting information, in short, to dumb down student writing.”
- This part is actually the main issue. Rather than focusing on just making blanket claims and pushing everything aside, humanreaders.org or any such initiatives should understand the inevitability of automated assessment and focus on how to combine it with human evaluation and other means better!

Those that rather made me smile, although I can’t brush aside these things as impossible

“students are subjected to a high-stakes response to their writing by a device that, in fact, cannot read, as even testing firms admit.”

“in machine-scored testing, often students falsely assume that their writing samples will be read by humans with a human’s insightful understanding”

“teachers are coerced into teaching the writing traits that they know the machine will count .. .. and into not teaching the major traits of successful writing.. .. “

- No I don’t have any specific comments on these. But, in parts, this is very imaginative…and in parts, it is not entirely impossible.

Those that gave new questions:
“conversely, students who knowingly write for a machine are placed in a bind since they cannot know what qualities of writing the machine will react to positively or negatively, the specific algorithms being closely guarded secrets of the testing firms (Frank, 1992; Rubin & O’Looney, 1990)—a bind made worse when their essay will be rated by both a human and a machine”
-as if we know what humans expect! I thought I wrote very a good essay on Sri Sri in my SSC exam’s question “my favourite poet”… and I got less marks (compared to my past performances) in Telugu, of all things. Apparently, teachers in school and those external examiners did not think alike! (or probably the examiner was a SriSri hater!)

“machines also cannot measure authentic audience awareness”
- Who can? Can humans do that with fellow humans? I don’t think so. I know there are people who think I am dumb. There are also those who think I am smart. There are also those who think I am mediocre. Who is right?

Conclusion of this series:

Although I did not do a real research-like reading and this is not some peer-reviewed article series, I spent some time doing this and it has been an enriching experience in terms of the insights it provided on the field of automated assessment and its criticisms.

What I learnt are two things:
* Automated Assessment is necessary and could not be avoided in future (among several reasons, because of the sheer number of students compared to the number of trained evaluators)
* Overcoming the flaws of automated assessments and efficient ways to combine it with trained human evaluators is more important, realistic and challenging than just branding everything as rubbish.

Although the above two are rather obvious, humanreaders.org “findings” and the way such “findings” immedietly grab media attention convinced me about the above two things more than before!

(Continued from part-2. All parts can be seen here)
***

Finding 5: machines require artificial essays finished within very short time frames (20-45 minutes) on topics of which student writers have no prior knowledge (Bridgeman, Trapani, & Yigal, 2012; Cindy, 2007; Jones, 2006; Perelman, 2012b; Streeter, Psotka, Laham, & MacCuish, 2002; Wang, & Brown, 2008; Wohlpart, Lindsey, & Rademacher, 2008)

First, Second, Third, Fourth, 7th references were not freely accessible.

Fifth Report (just search with the title in google): The primary conclusion of this report is: “The automated grading software performed as well as the better instructors in both trials, and well enough to be usefully applied to military instruction. The lower reliabilities observed in these essay sets reflect different instructors applying different criteria in grading these assignments. The statistical models that are created to do automated grading are also limited by the variability in the human grades.”

6th Report
This report studied the correlation between the machines and human raters and concluded, contrary to all previous studies, that there is no significant correlation between them.

While this was an interesting conclusion, I had a few questions which I could not clarify better as those references couldn’t be found online (for free). Here are the questions I have.

1. I think all these Automated Scoring systems work well within their framework. Like, the GRE exam grading works typically for GRE exam like essays but not for say, scoring what I write in my blog. But, they all are really customizable to any such target-specific responses (The NYT article on EdX, which began this series of blog posts from me makes that clear. It needs 100 sample essays and it will learn how to score essays from 101th on, to put it simply.).. So, is it fair to test Intellimetric or whatever system, on some essays that are about something else…and from other domain?

2. As far as I remember, all those other reports used pearson’s correlation. This report used spearman’s correlation. Can we compare the numbers we get directly? Why cant both be included?

-so far except the 4th one, the others seemed to be more about how well the automated scores approximate human judgements (looking at all the abstracts, which are freely available to read!!)

Finding 6: in these short trivial essays, mere length becomes a major determinant of score by both human and machine graders (Chodorow & Burstein, 2004; Perelman, 2012b)

First Report:
I actually found this report to be a very interesting read, for several reasons. Some of them are mentioned below:

“Since February 1999, ETS has used e-rater as one of the two initial readers for the GMAT writing assessments, and, in this capacity, it has scored more than 1 million essays. E-rater’s scores either match or are within one point of the human reader scores about 96% of the time”
-I thought this is an amazing performance (especially with such a high percentage even after grading 1 million essays or more!). I also like this part where they compare the machine-human agreement with human-human agreement, which is why I again think that no single entity should be the sole-scorer (there should be either 2 humans, 2 differently built machines or machine-human combinations).

While I am not completely comfortable with an automated system being the sole scorer in the low-stakes high school exams etc, I have the same feeling about human evaluators too. I think bias can exist both in a human and in a machine (in a machine, because its intelligence is limited. It learns only what it sees). I guess at that level, you always have an option to go back and demand a re-evaluation?

Primary conclusion of this report was that: “In practical terms, e-rater01 differs from human readers by only a very small amount in exact agreement, and it is indistinguishable from human readers in adjacent agreement. But despite these similarities, human readers and e-rater are not the same. When length is removed, human readers share more variance than e-rater01 shares with HR.”
-I think this is what the humanreaders.org use to come up with this finding. I do not know how to interpret this particular finding although it disappoints me a bit. Yet, the ETS report made a very interesting read.

A second conclusion that this “finding” does not focus on, but I found interesting, was the effect of the test taker’s Native Language on the overall scores. “There is a main effect for language in the data of the current study. The mixed and Spanish groups have higher scores on average than the Arabic and Japanese. These
differences remain even when length is removed. For most of the prompts, e-rater01 shows the same pattern of differences across native language groups as HR, even with length differences partialed out. Future work should include additional language groups to see if these results generalize.”
- After the recent Native Language Identification shared task, I got really curious on this very topic – the interaction between the native language and the scores that the test takers get. I think this is something that I might need to study further, irrespective of the disappointments during these readings!

Again, only the first report (from ETS) is freely accessible. If the purpose of humanreaders.org was to actually convey their message to a “curious” commoner, I think they are repeatedly failing in this by providing such inaccessible references. ….

Finding 7: machines are not able to approximate human scores for essays that do fit real-world writing conditions; instead, machines fail badly in rating essays written in these situations (Bridgeman, Trapani, & Yigal, 2012; Cindy, 2007; Condon, 2013; Elliot, Deess, Rudniy, & Joshi, 2012; Jones, 2006; Perelman, 2012b; Powers, Burstein, Chodorow, Fowles, & Kukich, 2002; Streeter, Psotka, Laham, & MacCuish, 2002; Wang & Brown, 2008; Wohlpart, Lindsey, & Rademacher, 2008)

First report, second, third, fifth, sixth, seventh, tenth reports are not freely accessible (although looking at the abstracts, I did not get an impression that these reports are against automated scoring in any way!).

A version of 4th report is available. Even this did not give me an impression that it has anything against automated scoring as such.. although there is a critical analysis of where automated scoring fails and it ended with a call for better systems.

(8th and 9th reports are already discussed in Finding 5)

-Again, I understand that this is an issue … but I don’t understand how just avoiding automated scoring will solve it.

Finding 8: high correlations between human scores and machine scores reported by testing firms are achieved, in part, when the testing firms train the humans to read like the machine, for instance, by directing the humans to disregard the truth or accuracy of assertions (Perelman, 2012b), and by requiring both machines and humans to use scoring scales of extreme simplicity

- Like I said before, I could not find a freely accessible page for (Perelman, 2012b). It would have been great if some freely accessible report was provided as additional reading here… because this claim/finding is among those most intriguing ones. First, they say that the machines are bad. Now, they say even the humans are bad! Should we be evaluated at all or not?

Conclusion for this part:
… And so, although its getting more and more disappointing… I decided to continue with these reports further because, even if the analysis and those claims (of humanreaders.org) are flawed according to me, I find those “findings” (or whatever they are) worth pondering about. There may not be a real solution. But, some of these issues need some thinking on the part of people who are capable of doing it better (e.g., teachers and researchers in education, researchers in automated scoring).

(to be continued..)

Hi All,

Dont get panic after seeing this huge error message in your SharePoint 2010 page.

Usually when you restore a Database to the database related to sharepoint application.

In my case when I restore a database to the old db of SharePoint site it occureed. To see the exact error we need to do little changes in the web.config of ” Security Token Service Application”.

Location of Security Token Service Application.

1) Open IIS.

2) Click Sites Folder

3) Click SharePoint Web Services (Small Globe Icon)

4) There at bottom you can see Security Token Service Application.

5) Right click and click Explore. There Open its web.config file and do the changes.

<!– Behavior List: –>
<behaviors>
<serviceBehaviors>
<behavior name=”SecurityTokenServiceBehavior”>
<!– The serviceMetadata behavior allows one to enable metadata (endpoints, bindings, services) publishing.
This configuration enables publishing of such data over HTTP GET.
This does not include metadata about the STS itself such as Claim Types, Keys and other elements to establish a trust.
–>

<serviceMetadata httpGetEnabled=”true” />

<serviceDebug httpHelpPageEnabled=”true” includeExceptionDetailInFaults=”true” />

<!– Default WCF throttling limits are too low –>

<serviceThrottling maxConcurrentCalls=”65536″ maxConcurrentSessions=”65536″ maxConcurrentInstances=”65536″ />

</behavior>

</serviceBehaviors>

</behaviors>

పూర్తిగా చదవండి

పూర్తిగా చదవండి

Situation: While Publishing ASP.NET files onto the remote server there will be no visual studio tools like gacutil.exe. So we can get doubt how can we add assemblies into gac folder (C:\windows\assembly\) without using gac. Here is the solution.

Solution:

1) Go to local Security Policy in windows 7.

2) Goto Local Policies ——> Security Options—> User Account Control: Run All Administrators in Admin Approval Mode

UAC

3) By Default it will be Enabled . Disable it and Add the dlls into C:\windows\Assembly\

UAC Enable

4) After adding again enable it and enjoy !!!

వాణి గారు.. ఈ playlistలో తెలుగు ఫొటోషాప్ లెసన్స్ వీడియోలు ఉంటాయి చూడండి..http://www.youtube.com/playlist?list=PLeDx22eBiFkQkpKuINtkywdlGrRaj9sQP

namaskaram sridhar garu naa peru
vani naaku telugulo photoshop
nerchukovalani vundi ela?

ఆర్యా!
తెలుగులో కంప్యూటర్ విషయాలు అందిస్తున్నందుకు మీకు నా కృతజ్ఞతలు. మీ కృషి శ్లాఘనీయము. తెలుగు వారికి మీరు కంప్యూటర్ సాంకేతిక సూచి పితామహుడు అనుటలో అతిశయోక్తి లేదని నా అభిప్రాయము. ఇలాగే మీరు మన తెలుగు వారికి మార్గ దర్శకులుగా ఉండాలను కోరుచున్నాను.
MS office కు సంబందించి తెలుగులో ఎక్సెల్, వర్డ్, అక్సెస్ మొదలగు వానిని తెలుగులో వివరించు డెమోలు అందివ్వ వలసినదిగా కోరుచున్నాను. తెలుగు ఫాంట్స్ ను తెలుగు విజయము వెబ్ నుండి డౌన్ లోడ్ చేసుకున్నాను. ఇంకా ఏమైనా ఉంటే తెలుపవలసినదిగ కోరుచున్నాను.
ఇట్లు
బి. వి. వరప్రసాద రాజు.
కడప

ఉపయోగకరంగా ఉన్న ఈ వీడియో ఆసక్తికరంగా అందించినందులకు ధన్యవాదాలు. జీ-మైల్ మాత్రమే గాక మన ఇతర ఇ ఆస్తులను వారసులకు ఎలా అందచేయాలో వివరిస్తూ మరణానంతరం జీవించటం ఎట్లా? అనే వ్యాసం వ్రాసాను. ఆసక్తికలవారు ఈ కింది గొలుసులో చూడవచ్చును.
http://deeptidhaara.blogspot.in/2013/02/blog-post.html

చాలా మంది సాఫ్ట్వేర్ వాళ్ళకు కూడా హార్డ్‌వేర్, పిసి అసెంబ్లింగ్, నెట్‌వర్కింగ్, లైనక్స్,సోషల్ మార్కెటింగ్ం పేజ్‌మేకర్, ఫొటోషాప్, కోరల్ డ్రా, ప్రీమియర్, ఇల్లస్ట్రేటర్, ఆఫీస్ వంటి వాటిలో కనీస పరిజ్ఞానం లేకపోవడం తప్పనిసరిగా ఒప్పుకోవలసిన విషయమే.
“మీకన్నీ తెలిసి ఉండొచ్చు…. చాలామందికి చాలా తెలియాల్సి ఉంది….” అన్నదానికి అందరు ఏఖీభవించాల్సిందే.ఎందుకంటే అది నిజం కాబట్టి.

//మీకన్నీ తెలిసి ఉండొచ్చు…. చాలామందికి చాలా తెలియాల్సి ఉంది….//

శ్రీధర్ గారు మీరు చెప్పింది నిజం.అడ్వాన్స్‌గా ఉండే కొందరికి మీరిచ్చే టిప్స్ సిల్లిగా అనిపించవచ్చు కాని బేసిక్ నాలెడ్జ్ లేని వారి గురించి వారు ఆలోచించరు అంతేకాకుండ తమకు తెలిసిన దాన్ని ఇతరులకు పంచరు, పంచే వారిని చూస్తే సహించరు ఇది లోక నీతి ఏం చేస్తాం ఖర్మ.

శ్రీధర్ గారికి నమస్కారం , మీ తదుపరి పత్రికలో ఇన్వెర్తెర్స్ మరియు సొలర్ ఇన్వెర్తెర్స్ గురించి ఒక అర్తికల్ రాయండి ప్రస్తుత కరెంట్ కోతల సమయంలో అందరికీ పనికివస్తుంది..

రంగా గారు నెట్‌బుక్ మామూలు లాప్ టాప్, డెస్క్ టాప్ కంప్యూటర్ చేసే అన్ని పనులూ చేయగలుగుతుంది. విండోస్, జావా, ఫ్లాష్ ప్లేయర్, Adobe AIR వంటి ఒక మామూలు పిసిలో వాడే అన్నీ వాడుకోవచ్చు.

అవగాహన లేని వారు అలా నెట్ బుక్ ని టాబ్లెట్ లా disabled డివైజ్ గా పోల్చి చూడడం నా దృష్టికీ పలుమార్లు వచ్చింది.

శ్రీధర్ గారూ,
నేను, స్టాక్ మార్కెట్ గ్రాఫ్ ఉండే వెబ్ సైట్ లు ఫాలో అవుతాను. సరే, టాబ్ చేతిలో ఉంటే మన పని సులువు అవుతుందని, టాబ్ కొందామని ఎంక్వైరీ చేస్తే టాబ్ వాటిని సపోర్ట్ చేయదని తెలిసి, నెట్ బుక్ కొందామని అనుకుంటున్నాను. అయితే, నెట్ బుక్ కూడా టాబ్ లా అన్నిటినీ సపోర్ట్ చేయదని కొంతమంది అన్నారు. మరి ఈ నెట్ బుక్ లో ఆ గ్రాఫ్ ల కోసం జావా మరియు adobe flash ప్లేయర్ లు install చేసుకోవచ్చా?

http://www.bazaartrend.com/nsecharts/intraday-free.php మీ నెట్ బుక్ లో ఈ లింక్ ఓపెన్ చేస్తే, NIFTY గ్రాఫ్ కనిపిస్తుందా?

దయచేసి రిప్లై ఇవ్వగలరు.

Sardar Jani Shaik ji, Thank you very much..