regression | Filome sharers have read the following articles about "regression" | www.filome.com filome.com http://www.filome.com/key/regression You're viewing shares 1-14 of 14 total shares for the keyword regression This is a keyword feed for "regression" from Filome read and shared items in Google Reader. If you would like to search or subscribe to category/keyword feeds for posts that are by shared with Google Reader users visit http://filome.com. en-us Copyright for these items belong to their original publishers. No filome, google reader, shared items, community knowledge organizer This is the keyword feed for "regression" from my read items in Google Reader. This is the keyword feed for "regression" from my read items in Google Reader. http://www.filome.com/images/feed_image.jpg regression | Filome sharers have read the following articles about "regression" | www.filome.com http://www.filome.com/key/regression This is a keyword feed for "regression" from Filome read and shared items in Google Reader. If you would like to search or subscribe to category/keyword feeds for posts that are by shared with Google Reader users visit http://filome.com. http://www.filome.com filome beta Psychoanalysis 101 http://www.psychologytoday.com/blog/apologies-freud/201006/psychoanalysis-101 Shared item Psychology Today Blogs
First shared by - Avi
syndication+ 0 | Search 1 | Shares 1

Ever since the Freuds (Sigmund and Anna), people have been throwing around lots of fancy-sounding terms like: narcissist, projection, and regression. What do these words mean? Here's a crash course and an explanation of some basic principles that apply in your day to day life:

Displacement: A psychological defense-oops better define defense-that involves transferring a feeling onto someone or something else.

Say the kids are sick. You catch it. Work is busy. Your husband is out of the country for business, and you want to stick your head in the oven. Only you can't. So, you do your very best juggling act. You get that big report in-but it is two days late and has a major typo relating to a material part of your thesis.

So where does displacement come in? As soon as your boss sees you, he tells you to, "get it together...Or else you can take that box of tissues and humidifier you dragged in from your sickbed and head straight for the unemployment office." Feeling as lousy emotionally as you do physically, you are overwrought. At this precise moment your husband calls, just to say hello, and you hear exotic music and glasses clinking in the background.

"What's that? Entertaining clients-on a cruise ship? Great, honey. I just coughed up a lung. Glad to hear you're having so much fun. Don't worry; Johnny didn't even notice you missed the soccer match, and Jenny will forgive you for missing her first home run. Why don't you give them a call?"

In just two seconds you've turned your spouse's smile to a frown. Just like that. And how do the subsequent calls go? Well he feels terrible and manages to say the wrong thing to your daughter. She hands the phone to her brother, telling him that he looks like a yeti. He wants to get away from his sister and acts rough with the family dog.

In short: Boss yells at wife, wife yells at husband, husband at daughter, daughter at son, and son kicks the dog. And all before breakfast-displacement is very quick and efficient. And it is the psychological equivalent of passing the buck.

Onto defense: a type of mental gymnastics your mind performs when it feels an uncomfortable or threatening feeling coming on. This goes on mostly out of your awareness--kind of like wearing a big protective mental shield all day long, every day. For example, you really want to watch the football game, and you are furious at your wife for inviting the annoying neighbors over for a barbeque-because instead of eating beef jerky and cheering-on your team, you're standing over a hot grill and refilling drinks all afternoon.

What defense do you use (unconsciously) to deal with your fury at your wife? Instead of realizing you are pissed off at her, you accuse her of being angry at you and you snarl at her to stop giving you such a hard time about the way you are flipping the burgers. In other words, you feel angry but perceive the situation incorrectly and conclude that the anger comes from your wife.

After you use the defense of projection, dishes begin to fly. The neighbors can't leave fast enough. You get to watch that ball game after all. Only you watch at the gym because you and your wife aren't speaking. Suddenly it doesn't matter who started it all-now everyone's angry.
So all of you serial projectors take note: when you are angry but that emotion feels uncomfortable, examine the situation closely to see what is really going on. If you can observe yourself and if you can realize that maybe, just maybe you incorrectly perceived yourself to be the recipient of another person's anger, you'll come out ahead of the game; the distortion you have become aware of is really a complex psychological event.

Regression: temporarily going backwards in time and even age-with no time machine involved; it's all psychological. Say you are in your therapist's office and find yourself preoccupied with the books on her shelf, the type of outgoing message she leaves on her voicemail, or the fact that she has a new haircut and you didn't expect her to change her appearance. It's like you are back to the feeling of being the scared little kid who is studying her mother before she goes out on the town. You are watching all of your mom's cues to make sure she'll come back just like she promised to do, because you miss her so much when she's away.

Regression is a clinical phenomenon that can occur everywhere, but is common in the therapist's office in the context of the therapeutic relationship. It can often be a very useful way to get at and work on important feelings from the past so they don't interfere with happiness and healthy functioning in the present.
Regression also applies to groups whose members all go back en masse to behaviors from times past. Don't believe me? Go to any preschool or elementary school during pick-up or drop-off time. Notice the cliques, the backstabbing, the immature behavior-and I don't mean the kids.

Which brings me to the final term, narcissist. Who doesn't hear this word uttered and bandied about on almost a daily basis? But what does narcissist mean? To whom does it refer? Surely it describes someone self-involved, self-centered, and rather puffed up. It also refers to someone who tends to take things personally. And it definitely refers to someone who is unable to step into the shoes of another; someone who cannot understand that the other is a separate human being with separate thoughts and feelings. Ringing any bells yet?

Most frenemies will fall into this category. Think of the friend who calls, constantly in crisis, always at 2 a.m. This friend is totally inured to the fact of-- insert any of the following: (a) you have a several months old baby and haven't had a second to yourself since Obama took office; (b) you have a very important meeting the next morning and you told this pal you need to sleep tonight; (c) it's the night before your wedding and you have, you know, stuff of your own on your mind; (d) you are on a great date that is going well and your friend is inconvenienced by your having a life; (e) all of the above. Frenemy, maybe. NARCISSIST? Definitely.

Next time you hear these terms at the water cooler or the coffee cart, you'll know just what they mean.

Many thanks to my great friend Dr. Wednesday Martin who made me write this post (that is known as externalization, by the way).



wife defense feeling psychological day wife defense yourself friend office feeling regression psychological narcissist husband ]]>Psychology Today Blogs
First shared by - Avi
syndication+ 0 | Search 1 | Shares 1

Ever since the Freuds (Sigmund and Anna), people have been throwing around lots of fancy-sounding terms like: narcissist, projection, and regression. What do these words mean? Here's a crash course and an explanation of some basic principles that apply in your day to day life:

Displacement: A psychological defense-oops better define defense-that involves transferring a feeling onto someone or something else.

Say the kids are sick. You catch it. Work is busy. Your husband is out of the country for business, and you want to stick your head in the oven. Only you can't. So, you do your very best juggling act. You get that big report in-but it is two days late and has a major typo relating to a material part of your thesis.

So where does displacement come in? As soon as your boss sees you, he tells you to, "get it together...Or else you can take that box of tissues and humidifier you dragged in from your sickbed and head straight for the unemployment office." Feeling as lousy emotionally as you do physically, you are overwrought. At this precise moment your husband calls, just to say hello, and you hear exotic music and glasses clinking in the background.

"What's that? Entertaining clients-on a cruise ship? Great, honey. I just coughed up a lung. Glad to hear you're having so much fun. Don't worry; Johnny didn't even notice you missed the soccer match, and Jenny will forgive you for missing her first home run. Why don't you give them a call?"

In just two seconds you've turned your spouse's smile to a frown. Just like that. And how do the subsequent calls go? Well he feels terrible and manages to say the wrong thing to your daughter. She hands the phone to her brother, telling him that he looks like a yeti. He wants to get away from his sister and acts rough with the family dog.

In short: Boss yells at wife, wife yells at husband, husband at daughter, daughter at son, and son kicks the dog. And all before breakfast-displacement is very quick and efficient. And it is the psychological equivalent of passing the buck.

Onto defense: a type of mental gymnastics your mind performs when it feels an uncomfortable or threatening feeling coming on. This goes on mostly out of your awareness--kind of like wearing a big protective mental shield all day long, every day. For example, you really want to watch the football game, and you are furious at your wife for inviting the annoying neighbors over for a barbeque-because instead of eating beef jerky and cheering-on your team, you're standing over a hot grill and refilling drinks all afternoon.

What defense do you use (unconsciously) to deal with your fury at your wife? Instead of realizing you are pissed off at her, you accuse her of being angry at you and you snarl at her to stop giving you such a hard time about the way you are flipping the burgers. In other words, you feel angry but perceive the situation incorrectly and conclude that the anger comes from your wife.

After you use the defense of projection, dishes begin to fly. The neighbors can't leave fast enough. You get to watch that ball game after all. Only you watch at the gym because you and your wife aren't speaking. Suddenly it doesn't matter who started it all-now everyone's angry.
So all of you serial projectors take note: when you are angry but that emotion feels uncomfortable, examine the situation closely to see what is really going on. If you can observe yourself and if you can realize that maybe, just maybe you incorrectly perceived yourself to be the recipient of another person's anger, you'll come out ahead of the game; the distortion you have become aware of is really a complex psychological event.

Regression: temporarily going backwards in time and even age-with no time machine involved; it's all psychological. Say you are in your therapist's office and find yourself preoccupied with the books on her shelf, the type of outgoing message she leaves on her voicemail, or the fact that she has a new haircut and you didn't expect her to change her appearance. It's like you are back to the feeling of being the scared little kid who is studying her mother before she goes out on the town. You are watching all of your mom's cues to make sure she'll come back just like she promised to do, because you miss her so much when she's away.

Regression is a clinical phenomenon that can occur everywhere, but is common in the therapist's office in the context of the therapeutic relationship. It can often be a very useful way to get at and work on important feelings from the past so they don't interfere with happiness and healthy functioning in the present.
Regression also applies to groups whose members all go back en masse to behaviors from times past. Don't believe me? Go to any preschool or elementary school during pick-up or drop-off time. Notice the cliques, the backstabbing, the immature behavior-and I don't mean the kids.

Which brings me to the final term, narcissist. Who doesn't hear this word uttered and bandied about on almost a daily basis? But what does narcissist mean? To whom does it refer? Surely it describes someone self-involved, self-centered, and rather puffed up. It also refers to someone who tends to take things personally. And it definitely refers to someone who is unable to step into the shoes of another; someone who cannot understand that the other is a separate human being with separate thoughts and feelings. Ringing any bells yet?

Most frenemies will fall into this category. Think of the friend who calls, constantly in crisis, always at 2 a.m. This friend is totally inured to the fact of-- insert any of the following: (a) you have a several months old baby and haven't had a second to yourself since Obama took office; (b) you have a very important meeting the next morning and you told this pal you need to sleep tonight; (c) it's the night before your wedding and you have, you know, stuff of your own on your mind; (d) you are on a great date that is going well and your friend is inconvenienced by your having a life; (e) all of the above. Frenemy, maybe. NARCISSIST? Definitely.

Next time you hear these terms at the water cooler or the coffee cart, you'll know just what they mean.

Many thanks to my great friend Dr. Wednesday Martin who made me write this post (that is known as externalization, by the way).



wife defense feeling psychological day wife defense yourself friend office feeling regression psychological narcissist husband ]]> Tue, 22 Jun 2010 16:00:57 -0400 30:00 tag:filome.com,1 Piwik 0.6.2 http://feedproxy.google.com/~r/PiwikReleases/~3/p8qDAkuHITw/ Shared item Piwik » Releases
First shared by - robdiana
syndication+ 0 | Search 1 | Shares 1

We are excited to announce the immediate availability of Piwik 0.6.2.

New features in this release

  • New report: Entry pages and Exit pages. You can now see for each page the number of entrances and exits, the bounce rate and exit rate.
  • In the Pages reports (Pages, Entry pages, Exit pages) you can now click on the icon to open the actual URL tracked in another tab.
  • Added CSV exports that import well in Excel: check out the new TSV (Excel) export link.
  • Added the complete set of favicons for search engines.

Tickets closed in this milestone

  • #306 Add entry / exit / time per page URL in the Actions > Pages report
  • #426 Google CSE should appear as Google CSE in search engines result
  • #541 How to pronounce Piwik?
  • #556 Add a link to the URL in the Actions > Pages report.
  • #950 "Loading data" position change not restored
  • #969 "Search" functionality doesn't apply to subtables
  • #1062 IE8 (native mode) regression: clicking on the table rows doesn't show the sub tables
  • #1153 when config file is not writable, display a warning message
  • #1235 Update to jquery 1.4.2 and jquery-ui 1.8.1
  • #1259 sum_time_spent is not always set in API responses for some rows
  • #1283 Bing image Search
  • #1300 (X)HTML elements don't have unique ids
  • #1322 Add a "CSV for Excel" download link
  • #1343 Fix missing translations in CoreHome
  • #1344 Fix missing translations in Goals plugin
  • #1345 Piwik One Click Update fails with bogus error if curl_exec times out
  • #1346 Re-opening Feedback form may display previous successfully sent or error message
  • #1347 Undefined index: excluded_parameters in Action.php
  • #1348 Google images not detected anymore
  • #1350 Adding missing search engine favicons
  • #1352 Fixing missing translation in MultiSites
  • #1354 Fix missing translations in title tag of several header.tpl
  • #1359 PHP 5.1.6 errors if EXPERIMENTAL_DATE_SUPPORT enabled
  • #1360 File integrity check regression first error discarded
  • #1364 Exclude by Cookie configureable cookie name and path
  • #1365 Update Google Icon
  • #1367 ExampleFeedburner broken
  • #1369 Visitor Generator generates Future Visits
  • #1376 "Ignore" cookie works only for current website
  • #1378 PHP 5.1.6 notices and errors
  • #1381 Add missing Google domains / URLs
  • #1383 Search engine update for 3721.com and Yahoo! China
  • #1384 Overrideable Content-Type for Piwik_View
  • #1387 Search engines: remove duplicate and dead links
#306 Add entry / exit / time per page URL in the Actions > Pages report
#426 Google CSE should appear as Google CSE in search engines result
#541 How to pronounce Piwik?
#556 Add a link to the URL in the Actions > Pages report.
#950 "Loading data" position change not restored
#969 "Search" functionality doesn't apply to subtables
#1062 IE8 (native mode) regression: clicking on the table rows doesn't show the sub tables
#1153 when config file is not writable, display a warning message
#1235 Update to jquery 1.4.2 and jquery-ui 1.8.1
#1259 sum_time_spent is not always set in API responses for some rows
#1283 Bing image Search
#1300 (X)HTML elements don't have unique ids
#1322 Add a "CSV for Excel" download link
#1343 Fix missing translations in CoreHome
#1344 Fix missing translations in Goals plugin
#1345 Piwik One Click Update fails with bogus error if curl_exec times out
#1346 Re-opening Feedback form may display previous successfully sent or error message
#1347 Undefined index: excluded_parameters in Action.php
#1348 Google images not detected anymore
#1350 Adding missing search engine favicons
#1352 Fixing missing translation in MultiSites
#1354 Fix missing translations in title tag of several header.tpl
#1359 PHP 5.1.6 errors if EXPERIMENTAL_DATE_SUPPORT enabled
#1360 File integrity check regression first error discarded
#1364 Exclude by Cookie configureable cookie name and path
#1365 Update Google Icon
#1367 ExampleFeedburner broken
#1369 Visitor Generator generates Future Visits
#1376 "Ignore" cookie works only for current website
#1378 PHP 5.1.6 notices and errors
#1381 Add missing Google domains / URLs
#1383 Search engine update for 3721.com and Yahoo! China
#1384 Overrideable Content-Type for Piwik_View
#1387 Search engines: remove duplicate and dead links


search missing pages google piwik ]]>Piwik » Releases
First shared by - robdiana
syndication+ 0 | Search 1 | Shares 1

We are excited to announce the immediate availability of Piwik 0.6.2.

New features in this release

  • New report: Entry pages and Exit pages. You can now see for each page the number of entrances and exits, the bounce rate and exit rate.
  • In the Pages reports (Pages, Entry pages, Exit pages) you can now click on the icon to open the actual URL tracked in another tab.
  • Added CSV exports that import well in Excel: check out the new TSV (Excel) export link.
  • Added the complete set of favicons for search engines.

Tickets closed in this milestone

  • #306 Add entry / exit / time per page URL in the Actions > Pages report
  • #426 Google CSE should appear as Google CSE in search engines result
  • #541 How to pronounce Piwik?
  • #556 Add a link to the URL in the Actions > Pages report.
  • #950 "Loading data" position change not restored
  • #969 "Search" functionality doesn't apply to subtables
  • #1062 IE8 (native mode) regression: clicking on the table rows doesn't show the sub tables
  • #1153 when config file is not writable, display a warning message
  • #1235 Update to jquery 1.4.2 and jquery-ui 1.8.1
  • #1259 sum_time_spent is not always set in API responses for some rows
  • #1283 Bing image Search
  • #1300 (X)HTML elements don't have unique ids
  • #1322 Add a "CSV for Excel" download link
  • #1343 Fix missing translations in CoreHome
  • #1344 Fix missing translations in Goals plugin
  • #1345 Piwik One Click Update fails with bogus error if curl_exec times out
  • #1346 Re-opening Feedback form may display previous successfully sent or error message
  • #1347 Undefined index: excluded_parameters in Action.php
  • #1348 Google images not detected anymore
  • #1350 Adding missing search engine favicons
  • #1352 Fixing missing translation in MultiSites
  • #1354 Fix missing translations in title tag of several header.tpl
  • #1359 PHP 5.1.6 errors if EXPERIMENTAL_DATE_SUPPORT enabled
  • #1360 File integrity check regression first error discarded
  • #1364 Exclude by Cookie configureable cookie name and path
  • #1365 Update Google Icon
  • #1367 ExampleFeedburner broken
  • #1369 Visitor Generator generates Future Visits
  • #1376 "Ignore" cookie works only for current website
  • #1378 PHP 5.1.6 notices and errors
  • #1381 Add missing Google domains / URLs
  • #1383 Search engine update for 3721.com and Yahoo! China
  • #1384 Overrideable Content-Type for Piwik_View
  • #1387 Search engines: remove duplicate and dead links
#306 Add entry / exit / time per page URL in the Actions > Pages report
#426 Google CSE should appear as Google CSE in search engines result
#541 How to pronounce Piwik?
#556 Add a link to the URL in the Actions > Pages report.
#950 "Loading data" position change not restored
#969 "Search" functionality doesn't apply to subtables
#1062 IE8 (native mode) regression: clicking on the table rows doesn't show the sub tables
#1153 when config file is not writable, display a warning message
#1235 Update to jquery 1.4.2 and jquery-ui 1.8.1
#1259 sum_time_spent is not always set in API responses for some rows
#1283 Bing image Search
#1300 (X)HTML elements don't have unique ids
#1322 Add a "CSV for Excel" download link
#1343 Fix missing translations in CoreHome
#1344 Fix missing translations in Goals plugin
#1345 Piwik One Click Update fails with bogus error if curl_exec times out
#1346 Re-opening Feedback form may display previous successfully sent or error message
#1347 Undefined index: excluded_parameters in Action.php
#1348 Google images not detected anymore
#1350 Adding missing search engine favicons
#1352 Fixing missing translation in MultiSites
#1354 Fix missing translations in title tag of several header.tpl
#1359 PHP 5.1.6 errors if EXPERIMENTAL_DATE_SUPPORT enabled
#1360 File integrity check regression first error discarded
#1364 Exclude by Cookie configureable cookie name and path
#1365 Update Google Icon
#1367 ExampleFeedburner broken
#1369 Visitor Generator generates Future Visits
#1376 "Ignore" cookie works only for current website
#1378 PHP 5.1.6 notices and errors
#1381 Add missing Google domains / URLs
#1383 Search engine update for 3721.com and Yahoo! China
#1384 Overrideable Content-Type for Piwik_View
#1387 Search engines: remove duplicate and dead links


search missing pages google piwik ]]> Fri, 28 May 2010 08:05:35 -0400 30:00 tag:filome.com,2 Interactive Flash Graphs and Charts For Your Website http://www.noupe.com/graphics/interactive-flash-graphs-and-charts-for-your-website.html Shared item Noupe
First shared by - robdiana
syndication+ 0 | Search 1 | Shares 1

By Usama Alam

Charts and graphs provide the visual representation of your data and help you facilitate the understanding of whatever it is you are trying to deliver. Whether you are publishing a survey report, an election result or a financial analysis, charts play a major role in data presentation.

Due to their interactivity, Flash-charts are particularly popular among designers and developers. These charts give users a way to interact with the chart, move charts among data values and see the data instead of reading, analyzing and visualizing it. In fact, many free and commercial chart plugins are available to download you can implement them on your website right away.

Free Charts

Astra Charts
Astra Charts is an open-source product of Yahoo! that enables you to present your tabular data in various graphical formats. Astra Charts empowers you to deliver animated flash charts on your websites and helps you customize and manipulate charts as per your needs. Astra Charts supports:

  • Bar Charts
  • Column Charts
  • Line Charts
  • Pie Charts

For developers, Yahoo! Developer Center offers complete ActionScript 3 class reference that gives you documentation of methods and properties used in Astra Charts.

Appfire FlashCharts Plugin
Aspire FlashCharts Plugin enables you to generate interactive and animated flash charts and dashboards on your web interface. Charts are controlled by macro parameters and xml data structures. Basic chart types include:

  • Area
  • Bar
  • Column
  • Pie & Donut
  • Funnel

Key features of Aspire FlashCharts Plugin include:

  • Animated and interactive
  • Custom labels and balloons
  • Manage charts using macros
  • Complete documentation, forum messages and blog entries

Google Chart Tools
The Google Chart Tools enable adding live charts to any web page. This library offers a rich gallery of visualizations provided as image charts (using a simple URL request to a Google chart server) or Interactive charts (using a Google developed JavaScript library). It can read live data from a variety of data sources. You can use bar charts, box charts, candlestick charts, compound charts, formulas, line charts, map charts, pie charts, QR codes, radar charts, scatter charts and Venn charts.

Open Flash Chart
Open Flash Chart is an open-source charting control that you can use to generate flash charts on your website. The project is primarily developed with PHP but server side libraries are available to download for Ruby, .NET, Perl, Java, Python, Coldfusion, Google Web Toolkit, Smalltalk, Pentaho and C. Chart types supported by Open Flash Chart are:

  • Line
  • Bar
  • Horizontal Bar
  • Stacked Bar
  • Candle
  • Area
  • Pie
  • Scatter
  • Radar

These charts can be further categorized as per your needs.

Key features of Open Flash Charts are:

  • Animated and interactive
  • Data exploration
  • Chart resizing
  • Tooltips
  • Save as image
  • Highlight points

FusionCharts Free
FusionCharts Free is an open-source charting component that enables you to integrate interactive and animated flash charts in your desktop and web applications. FusionChart Free supports all necessary chart types including:

  • 3D/2D Column & Bar Charts
  • Line Charts
  • 2D/3D Pie & Doughnut Charts
  • Stacked Bar
  • Area Charts
  • Stacked Charts
  • Combination Charts
  • Candlestick Charts
  • Funnel Chart
  • Gantt Chart

FusionCharts Free is cross platform and can be implemented with any server side technology like ASP, PHP, JSP, .NET, Python and even with simple HTML. With FusionCharts Free, you can also deliver interactive charts in PowerPoint presentations.

XML/SWF Charts
XML/SWF Charts offers a free tool to generate powerful flash charts to present your data graphically. It supports almost all necessary chart types such as:

  • Line Charts
  • Column Charts
  • Pie and Donut Charts
  • Stacked Charts
  • Area Charts
  • Candlestick Charts
  • Scattered Charts
  • Polar Charts
  • Bubble Charts
  • Mixed Charts

Key features of XML/SWF Charts include:

  • Generate charts with XML file
  • Use with any scripting language
  • Interactive scrolling and drill-down
  • Rich user experience with transparency, shadow, bevel, glow, blue and animated transitions
  • Full-screen mode
  • Save charts as bitmap
  • Unicode support to show special characters

Rich Chart Live
Rich Chart Live enables you to deliver flash charts on your website with minimal efforts. You don't need to download or install anything, it just needs a flash-enabled web browser. Rich Chart Live also offers a free edition that shows a logo with each chart showing the chart is generated with free edition of Rich Chart. It supports following chart types:

  • Area
  • Bar
  • Column
  • Pie & Donut
  • Line

Key features of Rich Chart Live include:

  • Platform independent
  • Animated charts
  • Generate using templates
  • Export data to flash or powerpoint
  • Supports Japenese, Chinese and other Unicode languages

Commercial Charts

Swiff Chart
Swiff Chart generates eye-catching animated charts with PHP, JSP and ASP.NET. It can query the database for the records and generates charts in flash, image and PDF formats. Supported chart types are:

  • Line
  • Bar
  • Area
  • Column
  • Pie & Donut
  • Gantt

Features of Swiff Chart include:

  • High impact graphics
  • Animated and interactive charts
  • Multiple chart types (Flash, SVG, PDF, Image)
  • Drill-down
  • Export to multiple formats including compressed file
  • Ready-made templates
  • Instant data import
  • Visual tools for customization
  • Custom Labels

Price:
Standard Edition: $149
Professional Edition: $299
Click here for complete pricing

Fly Charts
Fly Charts is an XML-based charting solution that enables you to generate interactive and animated charts on the fly. It doesn't need knowledge of flash, HTML or any other scripting language. Following chart types are supported by Fly Charts:

  • Line
  • Area
  • Column
  • Bar
  • Stair
  • Pie
  • Mixed

Fly Charts includes following features:

  • Animated and interactive charts
  • Generate charts on the fly
  • No technical knowledge required
  • Cross-platform
  • Rich and customizable interface
  • Chart creation using wizard

Price:
One Site: $99
Unlimited: $349
Developer Version: $799

AnyChart
AnyChart offers flash based charts for your desktop and web applications. It empowers your website with interactive charts for enhanced user experience. It is very easy to implement and works with almost all programming languages. Supported chart types are:

  • Column
  • Bar
  • Line
  • Spline
  • Area
  • Stacked
  • Dot/Marker
  • Bubble
  • Candlestick
  • Scatter
  • Pie & Donut
  • Gauge

Key features of AnyChart include:

  • Platform independent
  • Animated charts
  • Printable and exportable
  • Supports multiple axes
  • Supports logarithmic values
  • Inverted axes
  • Interactive dashboards
  • Custom fields, legends and tooltips
  • Multiple color techniques
  • Ready-made templates

Price:
Single Developer: $499
4 Developer Team License: $999
Unlimited Developer Site License: $2,999
Click here for complete pricing

Dundas Chart for .NET
Dundas Chart for .NET offers feature-rich charts and graphs for your ASP.NET and Windows applications. It supports chart animation in Flash and SVG formats. Following chart types are supported by Dundas Charts for .NET:

  • Column
  • Bar
  • Pie & Donut
  • Line
  • Area
  • Bubble/Points
  • Stock
  • Funnel & Pyramid
  • Error bar
  • Gantt
  • Spline
  • Circular

Key features of Dundas Charts for .NET include:

  • Animated flash and SVG charts
  • Automatic layout management
  • AJAX based server handling
  • Data drill-down
  • Multiple image formats
  • Multiple legends
  • Scrolling and zooming
  • Chart templates
  • View state management
  • Integrate with OLAP services
  • Callback events

Price:
Professional Edition: $769
Enterprise Edition: $3,299
Click here for complete pricing

amCharts
amCharts offers flash-based dynamic charts for your website. It can generate charts with data from CSV and XML and can also manipulate data from programming languages like PHP, .NET, ColdFusion, Perl, Java, and Ruby on Rails. amCharts offers a visual editor that you can use to configure your charts visually. It supports following chart types:

  • Column
  • Bar
  • Stock
  • Line
  • Area
  • Pie & Donut
  • Scatter
  • Bubble
  • Radar
  • Polar

Key features of amCharts include:

  • Platform independent
  • Exportable
  • Animated backgrounds
  • Reversed axis
  • Present logarithmic values
  • Custom description of data points
  • Drill-down
  • Gradient fills

Price:
Single Website License: 85 Euro
Multiple Websites License: 275 Euro
Developer License: 450 Euro
Click here for complete pricing

Flash Charts Pro
Flash Charts Pro offers cross-browser, platform-independent animated charts for your websites. It works with almost all scripting languages and processes data presented in XML format. Flash Charts Pro supports following chart types:

  • Column
  • Bar
  • Pie
  • Line
  • Candlestick
  • Stacked
  • Scatter

Key features of Flash Charts Pro include:

  • Interactive and animated charts
  • Multi-language support
  • Trendline and zone support
  • Dynamically resizable
  • Easy implementation
  • Inverted axes
  • Interactive dashboards
  • Custom fields, legends and tooltips
  • Multiple color techniques
  • Ready-made templates

Price:
User License: $59
Developer License: $99
Professional License: $199
Click here for complete pricing

Animated Chart
Animated Chart enables you to present static and dynamic flash-based animated charts on your website. The Pro edition enables you to deliver live data in your flash charts. Following chart types are supported by Animated Charts:

  • Column
  • Bar
  • Area
  • Bubble
  • Pie & Donut
  • Line
  • Scatter
  • Radar
  • Stock

Animated Chart includes following features:

  • Static and animated flash charts
  • Fully customizable
  • Visual effects like Appear, Grow, Fall, NeonAppear, Slide, Gather, Swing, Rotate
  • Apply Flash 8 animation effects
  • Export to HTML
  • Import data Excel or text file

Price:
Standard Edition: $49
Professional Edition: $69

Big Faceless Java Graph Library
Big Faceless Java Graph Library is a class library for Java to create 2D, 3D and shaded 3D interactive graphs on your website. It gives graphical representation in PNG, Flash, SVG or PDF formats. It supports following chart types:

  • Column
  • Bar
  • Area
  • Line
  • Pie & Donut
  • Scatter
  • Bubble
  • Funnel
  • Gantt

Key features of Big Faceless Java Graph Library include:

  • Easy implementation
  • Web service interface for graph creation
  • Adjustable light source
  • Generate charts in Flash, PNG, SVG and PDF formats
  • Create custom tags
  • Supports logarithmic axes
  • Complete documentation and API reference

Price:
Click here to calculate price as per your need

Rich Chart Builder
Rich Chart Builder enables you to generate animated and interactive flash charts for a variety of purposes. It also enables you to request custom chart for your specific needs. The distinguishing feature is chart can contain audio narration. Following chart types are supported by Rich Chart Builder:

  • Area
  • Bar
  • Column
  • Pie & Donut
  • Line

Key features of Rich Chart Builder include:

  • Animated and interactive charts
  • Platform independent
  • Audio narration
  • Publish to Powerpoint
  • Publish charts to image or PDF
  • Ready-made templates
  • Data export
  • Unicode support

Price:
Standard Edition: $95
Professional Edition: $199
Click here for complete pricing

WebCharts3D
WebCharts is a Java based data visualization solution for your web applications. It generates interactive and animated high quality charts in various formats including Flash. It incorporates sophisticated caching and resource management algorithms that enable the product to deliver high performance presentation. It includes following chart types:

  • Line
  • Bar
  • Pyramid
  • Step
  • Stair
  • Area
  • Curve
  • Scatter
  • Prism
  • Column
  • Box-whisker
  • Bubble
  • Ring Frame
  • Radar
  • Pie & Donut
  • Gauge
  • Gantt
  • Histogram & Regression
  • Maps

Features of WebCharts3D include:

  • Animated and interactive charts
  • Multiple chart types (Flash, SVG, PDF, Image)
  • Drill-down
  • High performance
  • Designer generated code
  • Automatic and manual customization
  • Multiple scales
  • Custom Labels
  • Optional placeable data table
  • Manage mouse events

Price:
Professional Edition: $550
Enterprise Edition: $750
Click here for complete pricing

Further Resources

Following articles are suggested for further reading:

About the author

Muhammad Usama Alam is a software engineer and web developer with over 8 years of experience in designing and developing business solutions for enterprise.

You can follow him @ twitter.



charts chart flash animated data ]]>Noupe
First shared by - robdiana
syndication+ 0 | Search 1 | Shares 1

By Usama Alam

Charts and graphs provide the visual representation of your data and help you facilitate the understanding of whatever it is you are trying to deliver. Whether you are publishing a survey report, an election result or a financial analysis, charts play a major role in data presentation.

Due to their interactivity, Flash-charts are particularly popular among designers and developers. These charts give users a way to interact with the chart, move charts among data values and see the data instead of reading, analyzing and visualizing it. In fact, many free and commercial chart plugins are available to download you can implement them on your website right away.

Free Charts

Astra Charts
Astra Charts is an open-source product of Yahoo! that enables you to present your tabular data in various graphical formats. Astra Charts empowers you to deliver animated flash charts on your websites and helps you customize and manipulate charts as per your needs. Astra Charts supports:

  • Bar Charts
  • Column Charts
  • Line Charts
  • Pie Charts

For developers, Yahoo! Developer Center offers complete ActionScript 3 class reference that gives you documentation of methods and properties used in Astra Charts.

Appfire FlashCharts Plugin
Aspire FlashCharts Plugin enables you to generate interactive and animated flash charts and dashboards on your web interface. Charts are controlled by macro parameters and xml data structures. Basic chart types include:

  • Area
  • Bar
  • Column
  • Pie & Donut
  • Funnel

Key features of Aspire FlashCharts Plugin include:

  • Animated and interactive
  • Custom labels and balloons
  • Manage charts using macros
  • Complete documentation, forum messages and blog entries

Google Chart Tools
The Google Chart Tools enable adding live charts to any web page. This library offers a rich gallery of visualizations provided as image charts (using a simple URL request to a Google chart server) or Interactive charts (using a Google developed JavaScript library). It can read live data from a variety of data sources. You can use bar charts, box charts, candlestick charts, compound charts, formulas, line charts, map charts, pie charts, QR codes, radar charts, scatter charts and Venn charts.

Open Flash Chart
Open Flash Chart is an open-source charting control that you can use to generate flash charts on your website. The project is primarily developed with PHP but server side libraries are available to download for Ruby, .NET, Perl, Java, Python, Coldfusion, Google Web Toolkit, Smalltalk, Pentaho and C. Chart types supported by Open Flash Chart are:

  • Line
  • Bar
  • Horizontal Bar
  • Stacked Bar
  • Candle
  • Area
  • Pie
  • Scatter
  • Radar

These charts can be further categorized as per your needs.

Key features of Open Flash Charts are:

  • Animated and interactive
  • Data exploration
  • Chart resizing
  • Tooltips
  • Save as image
  • Highlight points

FusionCharts Free
FusionCharts Free is an open-source charting component that enables you to integrate interactive and animated flash charts in your desktop and web applications. FusionChart Free supports all necessary chart types including:

  • 3D/2D Column & Bar Charts
  • Line Charts
  • 2D/3D Pie & Doughnut Charts
  • Stacked Bar
  • Area Charts
  • Stacked Charts
  • Combination Charts
  • Candlestick Charts
  • Funnel Chart
  • Gantt Chart

FusionCharts Free is cross platform and can be implemented with any server side technology like ASP, PHP, JSP, .NET, Python and even with simple HTML. With FusionCharts Free, you can also deliver interactive charts in PowerPoint presentations.

XML/SWF Charts
XML/SWF Charts offers a free tool to generate powerful flash charts to present your data graphically. It supports almost all necessary chart types such as:

  • Line Charts
  • Column Charts
  • Pie and Donut Charts
  • Stacked Charts
  • Area Charts
  • Candlestick Charts
  • Scattered Charts
  • Polar Charts
  • Bubble Charts
  • Mixed Charts

Key features of XML/SWF Charts include:

  • Generate charts with XML file
  • Use with any scripting language
  • Interactive scrolling and drill-down
  • Rich user experience with transparency, shadow, bevel, glow, blue and animated transitions
  • Full-screen mode
  • Save charts as bitmap
  • Unicode support to show special characters

Rich Chart Live
Rich Chart Live enables you to deliver flash charts on your website with minimal efforts. You don't need to download or install anything, it just needs a flash-enabled web browser. Rich Chart Live also offers a free edition that shows a logo with each chart showing the chart is generated with free edition of Rich Chart. It supports following chart types:

  • Area
  • Bar
  • Column
  • Pie & Donut
  • Line

Key features of Rich Chart Live include:

  • Platform independent
  • Animated charts
  • Generate using templates
  • Export data to flash or powerpoint
  • Supports Japenese, Chinese and other Unicode languages

Commercial Charts

Swiff Chart
Swiff Chart generates eye-catching animated charts with PHP, JSP and ASP.NET. It can query the database for the records and generates charts in flash, image and PDF formats. Supported chart types are:

  • Line
  • Bar
  • Area
  • Column
  • Pie & Donut
  • Gantt

Features of Swiff Chart include:

  • High impact graphics
  • Animated and interactive charts
  • Multiple chart types (Flash, SVG, PDF, Image)
  • Drill-down
  • Export to multiple formats including compressed file
  • Ready-made templates
  • Instant data import
  • Visual tools for customization
  • Custom Labels

Price:
Standard Edition: $149
Professional Edition: $299
Click here for complete pricing

Fly Charts
Fly Charts is an XML-based charting solution that enables you to generate interactive and animated charts on the fly. It doesn't need knowledge of flash, HTML or any other scripting language. Following chart types are supported by Fly Charts:

  • Line
  • Area
  • Column
  • Bar
  • Stair
  • Pie
  • Mixed

Fly Charts includes following features:

  • Animated and interactive charts
  • Generate charts on the fly
  • No technical knowledge required
  • Cross-platform
  • Rich and customizable interface
  • Chart creation using wizard

Price:
One Site: $99
Unlimited: $349
Developer Version: $799

AnyChart
AnyChart offers flash based charts for your desktop and web applications. It empowers your website with interactive charts for enhanced user experience. It is very easy to implement and works with almost all programming languages. Supported chart types are:

  • Column
  • Bar
  • Line
  • Spline
  • Area
  • Stacked
  • Dot/Marker
  • Bubble
  • Candlestick
  • Scatter
  • Pie & Donut
  • Gauge

Key features of AnyChart include:

  • Platform independent
  • Animated charts
  • Printable and exportable
  • Supports multiple axes
  • Supports logarithmic values
  • Inverted axes
  • Interactive dashboards
  • Custom fields, legends and tooltips
  • Multiple color techniques
  • Ready-made templates

Price:
Single Developer: $499
4 Developer Team License: $999
Unlimited Developer Site License: $2,999
Click here for complete pricing

Dundas Chart for .NET
Dundas Chart for .NET offers feature-rich charts and graphs for your ASP.NET and Windows applications. It supports chart animation in Flash and SVG formats. Following chart types are supported by Dundas Charts for .NET:

  • Column
  • Bar
  • Pie & Donut
  • Line
  • Area
  • Bubble/Points
  • Stock
  • Funnel & Pyramid
  • Error bar
  • Gantt
  • Spline
  • Circular

Key features of Dundas Charts for .NET include:

  • Animated flash and SVG charts
  • Automatic layout management
  • AJAX based server handling
  • Data drill-down
  • Multiple image formats
  • Multiple legends
  • Scrolling and zooming
  • Chart templates
  • View state management
  • Integrate with OLAP services
  • Callback events

Price:
Professional Edition: $769
Enterprise Edition: $3,299
Click here for complete pricing

amCharts
amCharts offers flash-based dynamic charts for your website. It can generate charts with data from CSV and XML and can also manipulate data from programming languages like PHP, .NET, ColdFusion, Perl, Java, and Ruby on Rails. amCharts offers a visual editor that you can use to configure your charts visually. It supports following chart types:

  • Column
  • Bar
  • Stock
  • Line
  • Area
  • Pie & Donut
  • Scatter
  • Bubble
  • Radar
  • Polar

Key features of amCharts include:

  • Platform independent
  • Exportable
  • Animated backgrounds
  • Reversed axis
  • Present logarithmic values
  • Custom description of data points
  • Drill-down
  • Gradient fills

Price:
Single Website License: 85 Euro
Multiple Websites License: 275 Euro
Developer License: 450 Euro
Click here for complete pricing

Flash Charts Pro
Flash Charts Pro offers cross-browser, platform-independent animated charts for your websites. It works with almost all scripting languages and processes data presented in XML format. Flash Charts Pro supports following chart types:

  • Column
  • Bar
  • Pie
  • Line
  • Candlestick
  • Stacked
  • Scatter

Key features of Flash Charts Pro include:

  • Interactive and animated charts
  • Multi-language support
  • Trendline and zone support
  • Dynamically resizable
  • Easy implementation
  • Inverted axes
  • Interactive dashboards
  • Custom fields, legends and tooltips
  • Multiple color techniques
  • Ready-made templates

Price:
User License: $59
Developer License: $99
Professional License: $199
Click here for complete pricing

Animated Chart
Animated Chart enables you to present static and dynamic flash-based animated charts on your website. The Pro edition enables you to deliver live data in your flash charts. Following chart types are supported by Animated Charts:

  • Column
  • Bar
  • Area
  • Bubble
  • Pie & Donut
  • Line
  • Scatter
  • Radar
  • Stock

Animated Chart includes following features:

  • Static and animated flash charts
  • Fully customizable
  • Visual effects like Appear, Grow, Fall, NeonAppear, Slide, Gather, Swing, Rotate
  • Apply Flash 8 animation effects
  • Export to HTML
  • Import data Excel or text file

Price:
Standard Edition: $49
Professional Edition: $69

Big Faceless Java Graph Library
Big Faceless Java Graph Library is a class library for Java to create 2D, 3D and shaded 3D interactive graphs on your website. It gives graphical representation in PNG, Flash, SVG or PDF formats. It supports following chart types:

  • Column
  • Bar
  • Area
  • Line
  • Pie & Donut
  • Scatter
  • Bubble
  • Funnel
  • Gantt

Key features of Big Faceless Java Graph Library include:

  • Easy implementation
  • Web service interface for graph creation
  • Adjustable light source
  • Generate charts in Flash, PNG, SVG and PDF formats
  • Create custom tags
  • Supports logarithmic axes
  • Complete documentation and API reference

Price:
Click here to calculate price as per your need

Rich Chart Builder
Rich Chart Builder enables you to generate animated and interactive flash charts for a variety of purposes. It also enables you to request custom chart for your specific needs. The distinguishing feature is chart can contain audio narration. Following chart types are supported by Rich Chart Builder:

  • Area
  • Bar
  • Column
  • Pie & Donut
  • Line

Key features of Rich Chart Builder include:

  • Animated and interactive charts
  • Platform independent
  • Audio narration
  • Publish to Powerpoint
  • Publish charts to image or PDF
  • Ready-made templates
  • Data export
  • Unicode support

Price:
Standard Edition: $95
Professional Edition: $199
Click here for complete pricing

WebCharts3D
WebCharts is a Java based data visualization solution for your web applications. It generates interactive and animated high quality charts in various formats including Flash. It incorporates sophisticated caching and resource management algorithms that enable the product to deliver high performance presentation. It includes following chart types:

  • Line
  • Bar
  • Pyramid
  • Step
  • Stair
  • Area
  • Curve
  • Scatter
  • Prism
  • Column
  • Box-whisker
  • Bubble
  • Ring Frame
  • Radar
  • Pie & Donut
  • Gauge
  • Gantt
  • Histogram & Regression
  • Maps

Features of WebCharts3D include:

  • Animated and interactive charts
  • Multiple chart types (Flash, SVG, PDF, Image)
  • Drill-down
  • High performance
  • Designer generated code
  • Automatic and manual customization
  • Multiple scales
  • Custom Labels
  • Optional placeable data table
  • Manage mouse events

Price:
Professional Edition: $550
Enterprise Edition: $750
Click here for complete pricing

Further Resources

Following articles are suggested for further reading:

About the author

Muhammad Usama Alam is a software engineer and web developer with over 8 years of experience in designing and developing business solutions for enterprise.

You can follow him @ twitter.



charts chart flash animated data ]]> Tue, 30 Mar 2010 07:00:40 -0400 30:00 tag:filome.com,3 Olympic overfitting? http://languagelog.ldc.upenn.edu/nll/?p=2151 Shared item Language Log
First shared by - christomer
syndication+ 0 | Search 1 | Shares 1

According to William Heuslein, "The man who predicts the medals", Forbes Magazine, 1/19/2010

Daniel Johnson makes remarkably accurate Olympic medal predictions. But he doesn't look at individual athletes or their events. The Colorado College economics professor considers just a handful of economic variables to come up with his prognostications.

The result: Over the past five Olympics, from the 2000 Summer Games in Sydney through the 2008 Summer Games in Beijing, Johnson's model demonstrated 94% accuracy between predicted and actual national medal counts.

First question: what do you think it means to "demonstrate 94% accuracy between predicted and actual national medal counts"?

If you guessed that it means "the predicted number of medals matched the actual number of medals 94 times out of 100, I'm sorry, you're wrong.

The next sentence of the Forbes article suggests what it really means:

For gold medal wins, the correlation is 87%.

And we can confirm from reading one of the papers cited on Johnson's web site (Daniel Johnson and Ayfer Ali, "A Tale of Two Seasons: Participation and Medal Counts at the Summer and Winter Olympic Games", Social Science Quarterly 85(4) 2004) that "94% accuracy between predicted and actual national medal counts" means that "the vector of predicted numbers of medals by country has a correlation of r=0.94 with the actual numbers of medals by country". (N.B. the text between the double quotes there is my summary, not Johnson & Ali's wording. And if you're a bit rusty on what correlation actually means, the wikipedia page is not bad; or just keep in mind that if you can turn one sequence of numbers into another by some combination of adding constants and multiplying by constants, their correlation will be "100%".)

This still sounds like pretty impressive predicting, but it could be true without any of the medal-count numbers actually coinciding. I wonder what proportion of Forbes' readers understood that? In fairness, Heuslein did slip in that "correlation", but then at the bottom of the piece, he lists what he calls the "Accuracy rate of Johnson's predictions" for the summer and winter games from 2000 to 2008, which (for total medals) vary from "93%" to "95%".

Anyhow, how well did "the man who predicts the medals" do this time around?

Forbes gives Johnson's "In Depth: Medal Predictions for Vancouver". And now that the Olympics is over, we can compare them with the actual medal counts, as documented by the New York Times. So I entered the data into a table, and calculated the correlations using this trivial R script:

X <- read.table("Olympics.table", header=TRUE)
totalcor <- cor(X[,"PredictedTotal"],X[,"ActualTotal"])
goldcor <- cor(X[,"PredictedGold"], X[,"ActualGold"])

The correlation for total medals? 0.625. The correlation for gold medals? 0.279.

How did 94% and 87% turn into 63% and 28%?

I'm not sure, and I don't have time this morning to nail it down I've got to finish my laundry, and hike over to the train station to catch the 8:30 Regional Rail for NYC, where I'm giving a talk at noon. But four possible explanations come to mind, all of which might simultaneously be true:

(1) Scribal error. Maybe Forbes' list of Johnson's predictions is wrong, or the NYT list of medal totals is wrong, or I made a mistake copying the numbers into my table. If so, someone will probably tell us in the comments.

(2) Regression to the mean. Maybe Johnson's luck ran out, like a mutual-fund manager whose string of good years was based more good fortune rather than good information.

(3) False advertising. Maybe the 94% correlation only applies to Johnson's predictions if you include not only the 13 countries in the Forbes list, but also all the other countries in the world, most of whom can trivially be predicted to win no medals at all, or almost none. If so, then r=0.94 may not really be very impressive.

(4) Overfitting. Although the cited paper does claim "out of sample" predictions that is, they calculate the model parameters on historical data, and then looks at the fit to recent data which was not used in "training" the model it's possible that they made some adjustments in the model structure in order to get it to work well, and perhaps a different set of adjustments would be needed to make it work well for this year's data.

My prediction: some combination of (3) and (4). With respect to (3), note that just padding the medal vectors with 70 zeros brings the total correlation up from r=0.63 to r=0.91:

padding <- vector(mode="numeric",70)
totalcor1 <- cor(c(X[,"PredictedTotal"],padding),c(X[,"ActualTotal"],padding))

My tentative conclusion: this is more evidence that when an economist is talking about numbers, you should put your hand on your wallet.

And, of course, when a journalist is interpreting a press release about a technical paper, you may need both hands and some help from your friends to avoid getting intellectually mugged.

[Via Phil Birnbaum, "An economist predicts the Olympic medal standings", Sabermetric Research 2/18/2010]



medal medals johnson correlation x ]]>Language Log
First shared by - christomer
syndication+ 0 | Search 1 | Shares 1

According to William Heuslein, "The man who predicts the medals", Forbes Magazine, 1/19/2010

Daniel Johnson makes remarkably accurate Olympic medal predictions. But he doesn't look at individual athletes or their events. The Colorado College economics professor considers just a handful of economic variables to come up with his prognostications.

The result: Over the past five Olympics, from the 2000 Summer Games in Sydney through the 2008 Summer Games in Beijing, Johnson's model demonstrated 94% accuracy between predicted and actual national medal counts.

First question: what do you think it means to "demonstrate 94% accuracy between predicted and actual national medal counts"?

If you guessed that it means "the predicted number of medals matched the actual number of medals 94 times out of 100, I'm sorry, you're wrong.

The next sentence of the Forbes article suggests what it really means:

For gold medal wins, the correlation is 87%.

And we can confirm from reading one of the papers cited on Johnson's web site (Daniel Johnson and Ayfer Ali, "A Tale of Two Seasons: Participation and Medal Counts at the Summer and Winter Olympic Games", Social Science Quarterly 85(4) 2004) that "94% accuracy between predicted and actual national medal counts" means that "the vector of predicted numbers of medals by country has a correlation of r=0.94 with the actual numbers of medals by country". (N.B. the text between the double quotes there is my summary, not Johnson & Ali's wording. And if you're a bit rusty on what correlation actually means, the wikipedia page is not bad; or just keep in mind that if you can turn one sequence of numbers into another by some combination of adding constants and multiplying by constants, their correlation will be "100%".)

This still sounds like pretty impressive predicting, but it could be true without any of the medal-count numbers actually coinciding. I wonder what proportion of Forbes' readers understood that? In fairness, Heuslein did slip in that "correlation", but then at the bottom of the piece, he lists what he calls the "Accuracy rate of Johnson's predictions" for the summer and winter games from 2000 to 2008, which (for total medals) vary from "93%" to "95%".

Anyhow, how well did "the man who predicts the medals" do this time around?

Forbes gives Johnson's "In Depth: Medal Predictions for Vancouver". And now that the Olympics is over, we can compare them with the actual medal counts, as documented by the New York Times. So I entered the data into a table, and calculated the correlations using this trivial R script:

X <- read.table("Olympics.table", header=TRUE)
totalcor <- cor(X[,"PredictedTotal"],X[,"ActualTotal"])
goldcor <- cor(X[,"PredictedGold"], X[,"ActualGold"])

The correlation for total medals? 0.625. The correlation for gold medals? 0.279.

How did 94% and 87% turn into 63% and 28%?

I'm not sure, and I don't have time this morning to nail it down I've got to finish my laundry, and hike over to the train station to catch the 8:30 Regional Rail for NYC, where I'm giving a talk at noon. But four possible explanations come to mind, all of which might simultaneously be true:

(1) Scribal error. Maybe Forbes' list of Johnson's predictions is wrong, or the NYT list of medal totals is wrong, or I made a mistake copying the numbers into my table. If so, someone will probably tell us in the comments.

(2) Regression to the mean. Maybe Johnson's luck ran out, like a mutual-fund manager whose string of good years was based more good fortune rather than good information.

(3) False advertising. Maybe the 94% correlation only applies to Johnson's predictions if you include not only the 13 countries in the Forbes list, but also all the other countries in the world, most of whom can trivially be predicted to win no medals at all, or almost none. If so, then r=0.94 may not really be very impressive.

(4) Overfitting. Although the cited paper does claim "out of sample" predictions that is, they calculate the model parameters on historical data, and then looks at the fit to recent data which was not used in "training" the model it's possible that they made some adjustments in the model structure in order to get it to work well, and perhaps a different set of adjustments would be needed to make it work well for this year's data.

My prediction: some combination of (3) and (4). With respect to (3), note that just padding the medal vectors with 70 zeros brings the total correlation up from r=0.63 to r=0.91:

padding <- vector(mode="numeric",70)
totalcor1 <- cor(c(X[,"PredictedTotal"],padding),c(X[,"ActualTotal"],padding))

My tentative conclusion: this is more evidence that when an economist is talking about numbers, you should put your hand on your wallet.

And, of course, when a journalist is interpreting a press release about a technical paper, you may need both hands and some help from your friends to avoid getting intellectually mugged.

[Via Phil Birnbaum, "An economist predicts the Olympic medal standings", Sabermetric Research 2/18/2010]



medal medals johnson correlation x ]]> Mon, 01 Mar 2010 07:01:14 -0500 30:00 tag:filome.com,4 Exploratory and confirmatory data analysis http://www.stat.columbia.edu/~cook/movabletype/archives/2010/02/exploratory_and.html Shared item Statistical Modeling, Causal Inference, and Social Science
First shared by - Ted_Louie
syndication+ 0 | Search 1 | Shares 1

Seth points me to this discussion he wrote on Tukey's famous book, Exploratory Data Analysis. I use exploratory methods all the time and have thought a lot about Tukey and his book and so wanted to add a few comments.

In particular, I'd like to separate Seth's important points about statistical practice from his inflammatory rhetoric and pop sociology. (Disclaimer: I engage in rantin', ragin', and pop sociology all the time--but, when I do it, it's for a higher purpose and it's ok.)

I have several important points to make here, so I recommend you stop whatever else you're doing and read all of this.

1. As Seth discusses, so-called exploratory and confirmatory methods are not in opposition (as is commonly assumed) but rather go together. The history on this is that "confirmatory data analysis" refers to p-values, while "exploratory data analysis" is all about graphs, but both these approaches are ways of checking models. I discuss this point more fully in my articles, Exploratory data analysis for complex models and A Bayesian formulation of exploratory data analysis and goodness-of-fit testing. The latter paper is particularly relevant for the readers of this blog, I think, as it discusses why Bayesians should embrace graphical displays of data--which I interpret as visual posterior predictive checks--rather than, as is typical, treating exploratory data analysis as something to be done quickly before getting to the real work of modeling.

2. Let me expand upon this point. Here's how I see things usually going in a work of applied statistics:

Step 1: Exploratory data analysis. Some plots of raw data, possibly used to determine a transformation.

Step 2: The main analysis--maybe model-based, maybe non-parametric, whatever. It is typically focused, not exploratory.

Step 3: That's it.

I have a big problem with Step 3 (as maybe you could tell already). Sometimes you'll also see some conventional model checks such as chi-squared tests or qq plots, but rarely anything exploratory. Which is really too bad, considering that a good model can make exploratory data analysis much more effective and, conversely, I'll understand and trust a model a lot more after seeing it displayed graphically along with data.

3. Seth writes:

A more accurate title of Tukey's book would have been Low-Status Data Analysis. Graphs and transformations are low-status. They are low-status because graphs are common and transformations are easy. Anyone can make a graph or transform their data. I believe they were neglected for that reason. To show their high status, statistics professors focused their research and teaching on more difficult and esoteric stuff -- like complicated regression. That the new stuff wasn't terribly useful (compared to graphs and transformations) mattered little. Like all academics -- like everyone -- they cared enormously about showing high status. It was far more important to be impressive than to be useful.

This is, in my experience, ridiculous. Seth doesn't just say that some work that is useful is low status, or that some work that is low status is useful. He says that useful statistical research work is generally low status. No, no, no, no! It's hard to be useful! Just about everybody in statistics tries to do work that is useful.

OK, I know what Seth is talking about. I used to teach at Berkeley (as did Seth), and indeed the statistics department back then was chock-full of high-status professors (the department was generally considered #1 or #2 in the world) who did little if anything useful in applied statistics. But they were trying to be useful! They were just so clueless that they didn't know better. And there was also some socialization going on, where the handful of faculty members who really were doing useful work seemed to most highly value the non-useful work of the others. It's certainly true that they didn't appreciate graphical methods or the challenges of getting down and dirty with data. (They might have dismissed such work as being insufficiently general and enduring, but in my experience such applied work has been crucial in motivating the development of new methods.) And they were also dismissive of applied research areas such as survey research that are fascinating and important but did not happen to be "hot" at the time. This is consistent with Seth's hypothesis of status-seeking, but I'm inclined to give the more charitable interpretation that my Berkeley colleagues wanted to work on what they viewed as the most important and challenging problems.

I repeat: I completely disagree with Seth's claim that, in statistics, it is "low status" to develop useful methods. Developing useful methods is as central to statistics as saving souls is in church--well, I'm guessing on this latter point, but stay with me on this, OK?--it's just hard to do, so some people occupy themselves in other useful ways such as building beautiful cathedrals (or managing bazaars). But having people actually use your methods--that's what it's all about.

Seth is the psychologist, not me, so I won't argue his claim that "everyone cares enormously about showing high status." In statistics, though, I think he has his status attributions backward. (I also worry about the circularity of arguments about status, like similar this-can-explain-anything arguments based on self-interest or, for that matter, "the unconscious.")

4. Hey, I almost forgot Seth's claim, "Anyone can make a graph or transform their data." No, no, no. Anyone can run a regression or an Anova! Regression and Anova are easy. Graphics is hard. Maybe things will change with the software and new media--various online tools such as Gapminder make graphs that are far far better than the Excel standard, and, with the advent of blogging, hot graphs are popular on the internet. We've come a long way from the days in which graphs were in drab black-and-white, when you had to fight to get them into journals, and when newspaper graphics were either ugly or (in the case of USA Today) of the notoriously trivial, "What are We Snacking on Today?", style.

Even now, though, if you're doing research work, it's much easier to run a plausible regression or Anova than to make a clear and informative graph. I'm an expert on this one. I've published thousands of graphs but created tens of thousands more that didn't make the cut.

One problem, perhaps, is that statistics advice is typically given in terms of the one correct analysis that you should do in any particular setting. If you're in situation A, do a two-sample t-test. In situation B, it's Ancova; for C you should do differences-in-differences; for D the correct solution is weighted least squares, and so forth. If you're lucky, you'll get to make a few choices regarding selection of predictors or choice of link function, but that's about it. And a lot of practical advice on statistics actually emphasizes how little choice you're supposed to have--the idea that you should decide on your data analysis before gathering any data, that it's cheating to do otherwise.

One of the difficulties with graphs is that it clearly doesn't work that way. Default regressions and default Anovas look like real regressions and Anovas, and in many cases they actually are! Default graphics may sometimes do a solid job at conveying information that you already have (see, for example, the graphs of estimated effect sizes and odds ratios that are, I'm glad to say, becoming standard adjuncts to regression analyses published in medical and public health journals), but it usually takes a bit more thought to really learn from a graph. Even the superplot--a graph I envisioned in my head back in 2003 (!) back at the very start of our Red State, Blue State project, before doing any data analysis at all--even the superplot required a lot of tweaking to look just right.

Perhaps things will change. One of my research interests is to more closely tie graphics to modeling and to develop a default process for looking through lots of graphs in a useful way. Researchers were doing this back in the 1960s and 70s--methods for rotating point clouds on the computer, and all that--but I'm thinking of something slightly different, something more closely connected to fitted models. But right now, no, graphs are harder, not easier, than formal statistical analysis.

Seth also writes:

Most statistics professors and their textbooks have neglected all uses of graphs and transformations, not just their exploratory uses. I used to think exploratory data analysis (and exploratory science more generally) needed different tools than confirmatory data analysis and confirmatory science. Now I don't. A big simplification.

All I can say is: Things are changing! The most popular book on Bayesian statistics and the most popular book on multilevel modeling have a strong exploratory framework and strongly support the view that similar tools are used for exploratory and confirmatory data analysis. (Not exactly the same tools, of course: there are lots of technical issues specific to graphics, and other technical issues specific to probability calculations. But I agree with Seth, there's a lot of overlap.)

5. To return briefly to Tukey's extremely influential book: EDA was published in 1977 but I believe he began to work in that area in the 1960s, about ten or fifteen years after doing his also extremely influential work on multiple comparisons (that is, confirmatory data analysis). I've always assumed that Tukey was finding p-values to be too limited a tool for doing serious applied statistics--something like playing the piano with mittens. I'm sure Tukey was super-clever at using the methods he had to learn from data, but it must have come to him that he was getting the most from his graphical displays of p-values and the like, rather than from their Type 1 and Type 2 error probabilities that he'd previously focused so strongly on. From there it was perhaps natural to ditch the p-values and the models entirely--as I've written before, I think Tukey went a bit too far in this particular direction--and see what he could learn by plotting raw data. This turned out to be an extremely fruitful direction for researchers, and followers in the Tukey tradition--I'm thinking of statisticians such as Bill Cleveland, Howard Wainer, Andreas Buja, Diane Cook, Antony Unwin, etc.--are continuing to make progress here.

(I'll have to talk with my colleagues who knew Tukey to see how accurate the above paragraph is as a description of his actual progression of thoughts, rather than merely my rational reconstruction.)

The actual methods and case studies in the EDA book . . . well, that's another story. Hanging rootograms, stem-and-leaf plots, goofy plots of interactions, the January temperature in Yuma, Nevada--all of this is best forgotten or, at best, remembered as an inspiration for important later work. Tukey was a compelling writer, though--I'll give him that. I read Exploratory Data Analysis twenty-five years ago and was captivated. At some point I escaped its spell and asked myself why I should care about the temperature in Yuma--but, at the time, it all made perfect sense. Even more so once I realized that his methods are ultimately model-based and can be even more effective if understood in that way (a point that I became dimly aware of while completing my Ph.D. thesis in 1990--when I realized that the model I'd spent two years working on didn't actually fit my data--and which I first formalized at a conference talk in 1997 and published in 2003 and 2004. It's funny how slowly these ideas develop.).

P.S. It's funny that Seth characterizes Freakonomics as low-status economics. Freakonomics is great, but this particular "rogue economist" was tenured at the University of Chicago and had been given major awards by the economics profession. The problem here, I think, is Seth's tendency to characterize everyone and everything into good guys and bad guys. Levitt's a good guy and low-status work is good, therefore Levitt's work must be low-status. Levitt's "real innovation" (in Seth's words) was to do excellent, headline-worthy work, then to actually get the headlines, to do more excellent, newsworthy work, and to attract the attention of a dedicated journalist. (Now he has to decide where to go next.)

That said, the economics profession (and academia in general) faces some tough questions, such as: How much in the way of resources should go toward studying sumo wrestlers and how much should go toward studying international trade? I assume Seth would argue that sumo wrestling is low-status and deserves more resources, while the study of international trade is overvalued and should be studied less. This line of thinking can run into trouble, though. For example, consider various declining academic subjects such as anthropology, philosophy, and classics. I'm not quite sure how Seth (or others) define status, but it's my impression that--in addition to their difficulties with funding and enrollment--anthropology, philosophy, and classics are lower status than more applied subjects such as economics, architecture, and law. I guess what I'm saying is: Low status isn't always such a good thing! Some topics are low-status for a good reason, for example that they've been bypassed or discredited. Consider various forgotten fads such as astrology or deconstructionism.

(I have some sympathy for "let the market decide" arguments: we don't need a commisar to tell us how many university appointments should be made in art history and how many in statistics, for example. Still, even in market or quasi-market situation, someone has to decide. For example, suppose a particular economics department has the choice of hiring a specialist in sumo wrestling or in international trade. They still have to make a decision.)

P.P.S. You might wonder why I'm spilling so much ink (so to speak) responding to Seth. I'll give two reasons (in addition to that he's my friend, and one of the functions of this blog is to allow me to share thoughts that would otherwise be restricted to personal emails). First is the journalistic tradition of the hook to hang a story on. All these thoughts spilled out of me, but Seth's thoughts were what got me started in this particular instance. Second, I know that exploratory statistical ideas have been important in Seth's own applied research, and so his views on what makes these methods work are worth listening to, even if I disagree on many points.

In any case, I'll try at some point to figure out a way to package these and related ideas in a more coherent way and perhaps publish them in some journal that no one reads or put them deep inside a book that nobody ever reads past chapter 5. Blogging as the higher procrastination, indeed.

P.P.P.S. Sometimes people ask me how much time I spend on blogging, and I honestly answer that I don't really know, because I intersperse blogging with other work. This time, however, I know the answer because I wrote this on the train. It took 2 hours.



data work status seth analysis ]]>Statistical Modeling, Causal Inference, and Social Science
First shared by - Ted_Louie
syndication+ 0 | Search 1 | Shares 1

Seth points me to this discussion he wrote on Tukey's famous book, Exploratory Data Analysis. I use exploratory methods all the time and have thought a lot about Tukey and his book and so wanted to add a few comments.

In particular, I'd like to separate Seth's important points about statistical practice from his inflammatory rhetoric and pop sociology. (Disclaimer: I engage in rantin', ragin', and pop sociology all the time--but, when I do it, it's for a higher purpose and it's ok.)

I have several important points to make here, so I recommend you stop whatever else you're doing and read all of this.

1. As Seth discusses, so-called exploratory and confirmatory methods are not in opposition (as is commonly assumed) but rather go together. The history on this is that "confirmatory data analysis" refers to p-values, while "exploratory data analysis" is all about graphs, but both these approaches are ways of checking models. I discuss this point more fully in my articles, Exploratory data analysis for complex models and A Bayesian formulation of exploratory data analysis and goodness-of-fit testing. The latter paper is particularly relevant for the readers of this blog, I think, as it discusses why Bayesians should embrace graphical displays of data--which I interpret as visual posterior predictive checks--rather than, as is typical, treating exploratory data analysis as something to be done quickly before getting to the real work of modeling.

2. Let me expand upon this point. Here's how I see things usually going in a work of applied statistics:

Step 1: Exploratory data analysis. Some plots of raw data, possibly used to determine a transformation.

Step 2: The main analysis--maybe model-based, maybe non-parametric, whatever. It is typically focused, not exploratory.

Step 3: That's it.

I have a big problem with Step 3 (as maybe you could tell already). Sometimes you'll also see some conventional model checks such as chi-squared tests or qq plots, but rarely anything exploratory. Which is really too bad, considering that a good model can make exploratory data analysis much more effective and, conversely, I'll understand and trust a model a lot more after seeing it displayed graphically along with data.

3. Seth writes:

A more accurate title of Tukey's book would have been Low-Status Data Analysis. Graphs and transformations are low-status. They are low-status because graphs are common and transformations are easy. Anyone can make a graph or transform their data. I believe they were neglected for that reason. To show their high status, statistics professors focused their research and teaching on more difficult and esoteric stuff -- like complicated regression. That the new stuff wasn't terribly useful (compared to graphs and transformations) mattered little. Like all academics -- like everyone -- they cared enormously about showing high status. It was far more important to be impressive than to be useful.

This is, in my experience, ridiculous. Seth doesn't just say that some work that is useful is low status, or that some work that is low status is useful. He says that useful statistical research work is generally low status. No, no, no, no! It's hard to be useful! Just about everybody in statistics tries to do work that is useful.

OK, I know what Seth is talking about. I used to teach at Berkeley (as did Seth), and indeed the statistics department back then was chock-full of high-status professors (the department was generally considered #1 or #2 in the world) who did little if anything useful in applied statistics. But they were trying to be useful! They were just so clueless that they didn't know better. And there was also some socialization going on, where the handful of faculty members who really were doing useful work seemed to most highly value the non-useful work of the others. It's certainly true that they didn't appreciate graphical methods or the challenges of getting down and dirty with data. (They might have dismissed such work as being insufficiently general and enduring, but in my experience such applied work has been crucial in motivating the development of new methods.) And they were also dismissive of applied research areas such as survey research that are fascinating and important but did not happen to be "hot" at the time. This is consistent with Seth's hypothesis of status-seeking, but I'm inclined to give the more charitable interpretation that my Berkeley colleagues wanted to work on what they viewed as the most important and challenging problems.

I repeat: I completely disagree with Seth's claim that, in statistics, it is "low status" to develop useful methods. Developing useful methods is as central to statistics as saving souls is in church--well, I'm guessing on this latter point, but stay with me on this, OK?--it's just hard to do, so some people occupy themselves in other useful ways such as building beautiful cathedrals (or managing bazaars). But having people actually use your methods--that's what it's all about.

Seth is the psychologist, not me, so I won't argue his claim that "everyone cares enormously about showing high status." In statistics, though, I think he has his status attributions backward. (I also worry about the circularity of arguments about status, like similar this-can-explain-anything arguments based on self-interest or, for that matter, "the unconscious.")

4. Hey, I almost forgot Seth's claim, "Anyone can make a graph or transform their data." No, no, no. Anyone can run a regression or an Anova! Regression and Anova are easy. Graphics is hard. Maybe things will change with the software and new media--various online tools such as Gapminder make graphs that are far far better than the Excel standard, and, with the advent of blogging, hot graphs are popular on the internet. We've come a long way from the days in which graphs were in drab black-and-white, when you had to fight to get them into journals, and when newspaper graphics were either ugly or (in the case of USA Today) of the notoriously trivial, "What are We Snacking on Today?", style.

Even now, though, if you're doing research work, it's much easier to run a plausible regression or Anova than to make a clear and informative graph. I'm an expert on this one. I've published thousands of graphs but created tens of thousands more that didn't make the cut.

One problem, perhaps, is that statistics advice is typically given in terms of the one correct analysis that you should do in any particular setting. If you're in situation A, do a two-sample t-test. In situation B, it's Ancova; for C you should do differences-in-differences; for D the correct solution is weighted least squares, and so forth. If you're lucky, you'll get to make a few choices regarding selection of predictors or choice of link function, but that's about it. And a lot of practical advice on statistics actually emphasizes how little choice you're supposed to have--the idea that you should decide on your data analysis before gathering any data, that it's cheating to do otherwise.

One of the difficulties with graphs is that it clearly doesn't work that way. Default regressions and default Anovas look like real regressions and Anovas, and in many cases they actually are! Default graphics may sometimes do a solid job at conveying information that you already have (see, for example, the graphs of estimated effect sizes and odds ratios that are, I'm glad to say, becoming standard adjuncts to regression analyses published in medical and public health journals), but it usually takes a bit more thought to really learn from a graph. Even the superplot--a graph I envisioned in my head back in 2003 (!) back at the very start of our Red State, Blue State project, before doing any data analysis at all--even the superplot required a lot of tweaking to look just right.

Perhaps things will change. One of my research interests is to more closely tie graphics to modeling and to develop a default process for looking through lots of graphs in a useful way. Researchers were doing this back in the 1960s and 70s--methods for rotating point clouds on the computer, and all that--but I'm thinking of something slightly different, something more closely connected to fitted models. But right now, no, graphs are harder, not easier, than formal statistical analysis.

Seth also writes:

Most statistics professors and their textbooks have neglected all uses of graphs and transformations, not just their exploratory uses. I used to think exploratory data analysis (and exploratory science more generally) needed different tools than confirmatory data analysis and confirmatory science. Now I don't. A big simplification.

All I can say is: Things are changing! The most popular book on Bayesian statistics and the most popular book on multilevel modeling have a strong exploratory framework and strongly support the view that similar tools are used for exploratory and confirmatory data analysis. (Not exactly the same tools, of course: there are lots of technical issues specific to graphics, and other technical issues specific to probability calculations. But I agree with Seth, there's a lot of overlap.)

5. To return briefly to Tukey's extremely influential book: EDA was published in 1977 but I believe he began to work in that area in the 1960s, about ten or fifteen years after doing his also extremely influential work on multiple comparisons (that is, confirmatory data analysis). I've always assumed that Tukey was finding p-values to be too limited a tool for doing serious applied statistics--something like playing the piano with mittens. I'm sure Tukey was super-clever at using the methods he had to learn from data, but it must have come to him that he was getting the most from his graphical displays of p-values and the like, rather than from their Type 1 and Type 2 error probabilities that he'd previously focused so strongly on. From there it was perhaps natural to ditch the p-values and the models entirely--as I've written before, I think Tukey went a bit too far in this particular direction--and see what he could learn by plotting raw data. This turned out to be an extremely fruitful direction for researchers, and followers in the Tukey tradition--I'm thinking of statisticians such as Bill Cleveland, Howard Wainer, Andreas Buja, Diane Cook, Antony Unwin, etc.--are continuing to make progress here.

(I'll have to talk with my colleagues who knew Tukey to see how accurate the above paragraph is as a description of his actual progression of thoughts, rather than merely my rational reconstruction.)

The actual methods and case studies in the EDA book . . . well, that's another story. Hanging rootograms, stem-and-leaf plots, goofy plots of interactions, the January temperature in Yuma, Nevada--all of this is best forgotten or, at best, remembered as an inspiration for important later work. Tukey was a compelling writer, though--I'll give him that. I read Exploratory Data Analysis twenty-five years ago and was captivated. At some point I escaped its spell and asked myself why I should care about the temperature in Yuma--but, at the time, it all made perfect sense. Even more so once I realized that his methods are ultimately model-based and can be even more effective if understood in that way (a point that I became dimly aware of while completing my Ph.D. thesis in 1990--when I realized that the model I'd spent two years working on didn't actually fit my data--and which I first formalized at a conference talk in 1997 and published in 2003 and 2004. It's funny how slowly these ideas develop.).

P.S. It's funny that Seth characterizes Freakonomics as low-status economics. Freakonomics is great, but this particular "rogue economist" was tenured at the University of Chicago and had been given major awards by the economics profession. The problem here, I think, is Seth's tendency to characterize everyone and everything into good guys and bad guys. Levitt's a good guy and low-status work is good, therefore Levitt's work must be low-status. Levitt's "real innovation" (in Seth's words) was to do excellent, headline-worthy work, then to actually get the headlines, to do more excellent, newsworthy work, and to attract the attention of a dedicated journalist. (Now he has to decide where to go next.)

That said, the economics profession (and academia in general) faces some tough questions, such as: How much in the way of resources should go toward studying sumo wrestlers and how much should go toward studying international trade? I assume Seth would argue that sumo wrestling is low-status and deserves more resources, while the study of international trade is overvalued and should be studied less. This line of thinking can run into trouble, though. For example, consider various declining academic subjects such as anthropology, philosophy, and classics. I'm not quite sure how Seth (or others) define status, but it's my impression that--in addition to their difficulties with funding and enrollment--anthropology, philosophy, and classics are lower status than more applied subjects such as economics, architecture, and law. I guess what I'm saying is: Low status isn't always such a good thing! Some topics are low-status for a good reason, for example that they've been bypassed or discredited. Consider various forgotten fads such as astrology or deconstructionism.

(I have some sympathy for "let the market decide" arguments: we don't need a commisar to tell us how many university appointments should be made in art history and how many in statistics, for example. Still, even in market or quasi-market situation, someone has to decide. For example, suppose a particular economics department has the choice of hiring a specialist in sumo wrestling or in international trade. They still have to make a decision.)

P.P.S. You might wonder why I'm spilling so much ink (so to speak) responding to Seth. I'll give two reasons (in addition to that he's my friend, and one of the functions of this blog is to allow me to share thoughts that would otherwise be restricted to personal emails). First is the journalistic tradition of the hook to hang a story on. All these thoughts spilled out of me, but Seth's thoughts were what got me started in this particular instance. Second, I know that exploratory statistical ideas have been important in Seth's own applied research, and so his views on what makes these methods work are worth listening to, even if I disagree on many points.

In any case, I'll try at some point to figure out a way to package these and related ideas in a more coherent way and perhaps publish them in some journal that no one reads or put them deep inside a book that nobody ever reads past chapter 5. Blogging as the higher procrastination, indeed.

P.P.P.S. Sometimes people ask me how much time I spend on blogging, and I honestly answer that I don't really know, because I intersperse blogging with other work. This time, however, I know the answer because I wrote this on the train. It took 2 hours.



data work status seth analysis ]]> Tue, 16 Feb 2010 19:35:27 -0500 30:00 tag:filome.com,5 Suggestions for R books http://stackoverflow.com/questions/2270631/suggestions-for-r-books Shared item active questions tagged r - Stack Overflow
First shared by - Zaki_Manian
syndication+ 0 | Search 1 | Shares 1

I intended to write this post ages ago... When I started to use R, several books seemed to me like a Bible of R language. On top of all, they were (and still are) free (as in "free beer"):

  • An Introduction to R - W. N. Venables, D. M. Smith and the R Development Core Team
  • R for Beginners - Emmanuel Paradis
  • Using R for Data Analysis and Graphics: Introduction, Code and Commentary - J. H. Maindonald
  • simpleR: Using R for Introductory Statistics - John Verzani
  • The R Guide - W. J. Owen

Recently I acquainted with an interesting book and quite original approach to R :

  • The R Inferno - Patrick Burns

But there are a lot of good, non-free books:

  • A Beginner's Guide to R - Alain F. Zuur, Elena N. Ieno, Erik H.W.G. Meesters quite simple, excelent for beginners
  • A First Course in Statistical Programming with R - W. John Braun and Duncan J. Murdoch this one requires certain mathematical kills, but it's great, recommend
  • A Handbook of Statistical Analyses Using R - Brian S. Everitt and Ibrsten Hothorn
  • A Modern Approach to Regression with R - Simon J. Sheather
  • Analyzing Linguistic Data A Practical Introduction to Statistics Using R - R. H. Baayen
  • Applied Spatial Data Analysis with R - Roger S. Bivand, Edzer J. Pebesma and Virgilio Gmez-Rubio
  • Applied Statistics Using SPSS, STATISTICA, MATLAB and R - Joaquim P. Marques de S great book for those who are migrating frome SPSS and MATLAB to R
  • Bayesian Computation with R - Jim Albert good reference for social scientists
  • Data Analysis and Graphics Using R - an Example-Based Approach - John Maindonald, W. John Braun
  • Data Manipulation with R - Phil Spector recommend
  • Data Mashups in R - Jeremy Leipzig and Xiao-Yi Li demonstration of R's extensibility, concise and good reference
  • Dynamic Linear Models with R - Giovanni Petris, Sonia Petrone and Patrizia Campagnoli
  • Extending the Linear Model with R - Julian J. Faraway
  • ggplot2 - Elegant Graphics for Data Analysis - Hadley Wickham great one
  • Interactive and Dynamic Graphics for Data Analysis With R and GGobi - Dianne Cook and Deborah F. Swayne
  • Introduction to Applied Bayesian Statistics and Estimation for Social Scientists - Scott M. Lynch
  • Introduction to Scientific Programming and Simulation using R - Owen Jones, Robert Maillardet and Andrew Robinson
  • Introductory Statistics with R - Peter Dalgaard excelent for beginners
  • Lattice Multivariate Data Visualization with R - Deepayan Sarkar
  • Linear Models with R - Julian J. Faraway
  • Modern Applied Statistics with S - W. N. Venables and B. D. Ripley
  • Practical Regression and Anova using R - Julian J. Faraway
  • Practical Tools For Exploring Data and Models - Hadley Alexander Wickham [correct me if I'm wrong, but I think this one is free]
  • R for SAS and SPSS Users - Robert A. Muenchen great tool if you migrate from SPSS, as I did once upon a time...
  • R Graphics - Paul Murrell excelent book for learning low-level graph capabilities in R
  • R in a Nutshell: A Desktop Quick Reference - Joseph Adler great companion for programming in R
  • R in Action - Data Analysis and Graphics with R - Robert Kabacoff great book, even better author's site
  • Robust Statistical Methods with R - Jana Jurekov and Jan Picek
  • R Through Excel - Richard M. Heiberger and Erich Neuwirth great tool if you use MS Office
  • Software for Data Analysis - Programming with R - John M. Chambers recommend
  • Statistics and Data with R: An applied approach through examples - Yosef Cohen and Jeremiah Y. Cohen
  • Statistics: An Introduction using R - Michael J. Crawley similar to Daalgard's book
  • The R Book - Michael J. Crawley this is one of those R-bibles, thorough and complete R reference
  • Time Series Analysis and Its Applications With R Examples - Robert H. Shumway and David S. Stoffer
  • Time Series Analysis With Applications in R - Jonathan D. Cryer and Kung-Sik Chan
  • Wavelet Methods in Statistics with R - G. P. Nason

Now, my question/appel is:

Please post your favourite R books, and please suggest me some R books adequate for psychologists / social scientists!

Thank you!



r data analysis statistics book ]]>active questions tagged r - Stack Overflow
First shared by - Zaki_Manian
syndication+ 0 | Search 1 | Shares 1

I intended to write this post ages ago... When I started to use R, several books seemed to me like a Bible of R language. On top of all, they were (and still are) free (as in "free beer"):

  • An Introduction to R - W. N. Venables, D. M. Smith and the R Development Core Team
  • R for Beginners - Emmanuel Paradis
  • Using R for Data Analysis and Graphics: Introduction, Code and Commentary - J. H. Maindonald
  • simpleR: Using R for Introductory Statistics - John Verzani
  • The R Guide - W. J. Owen

Recently I acquainted with an interesting book and quite original approach to R :

  • The R Inferno - Patrick Burns

But there are a lot of good, non-free books:

  • A Beginner's Guide to R - Alain F. Zuur, Elena N. Ieno, Erik H.W.G. Meesters quite simple, excelent for beginners
  • A First Course in Statistical Programming with R - W. John Braun and Duncan J. Murdoch this one requires certain mathematical kills, but it's great, recommend
  • A Handbook of Statistical Analyses Using R - Brian S. Everitt and Ibrsten Hothorn
  • A Modern Approach to Regression with R - Simon J. Sheather
  • Analyzing Linguistic Data A Practical Introduction to Statistics Using R - R. H. Baayen
  • Applied Spatial Data Analysis with R - Roger S. Bivand, Edzer J. Pebesma and Virgilio Gmez-Rubio
  • Applied Statistics Using SPSS, STATISTICA, MATLAB and R - Joaquim P. Marques de S great book for those who are migrating frome SPSS and MATLAB to R
  • Bayesian Computation with R - Jim Albert good reference for social scientists
  • Data Analysis and Graphics Using R - an Example-Based Approach - John Maindonald, W. John Braun
  • Data Manipulation with R - Phil Spector recommend
  • Data Mashups in R - Jeremy Leipzig and Xiao-Yi Li demonstration of R's extensibility, concise and good reference
  • Dynamic Linear Models with R - Giovanni Petris, Sonia Petrone and Patrizia Campagnoli
  • Extending the Linear Model with R - Julian J. Faraway
  • ggplot2 - Elegant Graphics for Data Analysis - Hadley Wickham great one
  • Interactive and Dynamic Graphics for Data Analysis With R and GGobi - Dianne Cook and Deborah F. Swayne
  • Introduction to Applied Bayesian Statistics and Estimation for Social Scientists - Scott M. Lynch
  • Introduction to Scientific Programming and Simulation using R - Owen Jones, Robert Maillardet and Andrew Robinson
  • Introductory Statistics with R - Peter Dalgaard excelent for beginners
  • Lattice Multivariate Data Visualization with R - Deepayan Sarkar
  • Linear Models with R - Julian J. Faraway
  • Modern Applied Statistics with S - W. N. Venables and B. D. Ripley
  • Practical Regression and Anova using R - Julian J. Faraway
  • Practical Tools For Exploring Data and Models - Hadley Alexander Wickham [correct me if I'm wrong, but I think this one is free]
  • R for SAS and SPSS Users - Robert A. Muenchen great tool if you migrate from SPSS, as I did once upon a time...
  • R Graphics - Paul Murrell excelent book for learning low-level graph capabilities in R
  • R in a Nutshell: A Desktop Quick Reference - Joseph Adler great companion for programming in R
  • R in Action - Data Analysis and Graphics with R - Robert Kabacoff great book, even better author's site
  • Robust Statistical Methods with R - Jana Jurekov and Jan Picek
  • R Through Excel - Richard M. Heiberger and Erich Neuwirth great tool if you use MS Office
  • Software for Data Analysis - Programming with R - John M. Chambers recommend
  • Statistics and Data with R: An applied approach through examples - Yosef Cohen and Jeremiah Y. Cohen
  • Statistics: An Introduction using R - Michael J. Crawley similar to Daalgard's book
  • The R Book - Michael J. Crawley this is one of those R-bibles, thorough and complete R reference
  • Time Series Analysis and Its Applications With R Examples - Robert H. Shumway and David S. Stoffer
  • Time Series Analysis With Applications in R - Jonathan D. Cryer and Kung-Sik Chan
  • Wavelet Methods in Statistics with R - G. P. Nason

Now, my question/appel is:

Please post your favourite R books, and please suggest me some R books adequate for psychologists / social scientists!

Thank you!



r data analysis statistics book ]]> Tue, 16 Feb 2010 13:30:30 -0500 30:00 tag:filome.com,6 Physiological politics http://languagelog.ldc.upenn.edu/nll/?p=2121 Shared item Language Log
First shared by - christomer
syndication+ 0 | Search 1 | Shares 1

Carl Zimmer, among other readers, has pointed me to the latest outbreak of bio-political punditry. This time it isn't David Brooks, but rather Nicholas Kristof, "Our Politics May Be All in Our Head", 2/13/2010:

We all know that liberals and conservatives are far apart on health care. But in the way their brains work? Even in automatic reflexes, like blinking? Or the way their glands secrete moisture?

That's the suggestion of some recent research. It hints that the roots of political judgments may lie partly in fundamental personality types and even in the hard-wiring of our brains.

Researchers have found, for example, that some humans are particularly alert to threats, particularly primed to feel vulnerable and perceive danger. Those people are more likely to be conservatives.

Kristof's "recent research" link goes to Kevin Smith, Douglas Oxley, Matthew Hibbing, John Alford, and John Hibbing, "The Ick Factor: Disgust Sensitivity as a Predictor of Political Attitudes". The abstract:

Mounting evidence suggests political attitudes connect to broad, dispositional, perhaps biological temperaments. We add to these empirical results by reporting a correlation between certain political attitudes and physiological reactions to disgusting stimuli such as human excrement and worms being eaten. Specifically, we find that individuals whose skin conductance levels increase when viewing disgusting images are more likely to adopt conservative positions on homosexuality and perhaps a broader range of sex and reproduction issues, a result that serves as a useful illustration of the larger point that political attitudes cannot be separated from generic physiological traits.

This is listed as a "Paper prepared for presentation at the annual meeting of the Midwest Political Science Association", April 2008; but unfortunately, the copy available at the link given is almost impossible to read (because the type is so small, with no method that I could find to enlarge it), and I couldn't find another copy on line. However, a similar set of ideas and experiments are presented in a more accessible way in Douglas Oxley, et al., "Political Attitudes Vary with Physiological Traits", Science 321:5896, 9/19/2008:

Although political views have been thought to arise largely from individuals' experiences, recent research suggests that they may have a biological basis. We present evidence that variations in political attitudes correlate with physiological traits. In a group of 46 adult participants with strong political beliefs, individuals with measurably lower physical sensitivities to sudden noises and threatening visual images were more likely to support foreign aid, liberal immigration policies, pacifism, and gun control, whereas individuals displaying measurably higher physiological reactions to those same stimuli were more likely to favor defense spending, capital punishment, patriotism, and the Iraq War. Thus, the degree to which individuals are physiologically responsive to threat appears to indicate the degree to which they advocate policies that protect the existing social structure from both external (outgroup) and internal (norm-violator) threats.

In other words, the Science paper deals with reactions to sudden noises and theatening visual images, correlated with political attitudes toward level of support for what the authors call "protective policies". The nearly-unreadable conference paper makes an analogous argument about reactions to disgusting images correlated with political attitudes towards "sex and reproduction issues", especially homosexuality. So I'm going to discuss the Science paper if someone can point me to a readable copy of the "disgust" paper, I'll be happy to apply the same analysis there.

(By the way, you can see from the quoted abstracts that Kristof's account of this work is not very different from what the authors themselves have to say about it. This is not a case of egregious journalistic misunderstanding or over-interpretation.)

In the rest of this post, I'm going to go over some aspects of the analysis in the Science paper, and argue that the explanation offered is at least excessive and possibly wrong. I'm somewhat hampered in doing this by the fact that some relevant aspects of the authors' data are not published, and also by (what I found to be) some ambiguities in the presentation of their data and their models. If as a result I've misunderstand what they did or how they interpreted it, I apologize in advance. I plan to write to the authors and ask for clarification and for a copy of their original data, which would allow me (and others) to make a more full and fair evaluation.

First, who were the subjects?

Subjects were recruited in May of 2007 by the Bureau of Sociological Research at the University of Nebraska-Lincoln (BOSR). BOSR contacted a random sample of residents of Lincoln, Nebraska. This initial telephone call followed an introductory letter and was used to pose a limited number of items to respondents with the intention of obtaining a group of individuals with strong political convictions toward whom intense and more focused investigation would be directed. The following three questions were used to screen potential subjects. Yes or no response categories were given for all three questions.

1. Do you follow politics or political issues closely?

2. Is there a certain political issue or set of political issues you feel strongly about?

3. Have you ever supported a particular political issue or cause?

A total of 1310 people were contacted and 608 of them completed the screening items. Subjects were recruited for this particular project only if they responded yes to all three questions. A total of 143 respondents did so and were agreeable in principle to coming to the lab. BOSR was able to schedule (and secure the attendance of) 48 individuals at both sessions (survey and physiological) of this project. Health problems rendered the data from one individual unusable and mechanical problems with a sensor spoiled the data from another, leaving a final total of 46 participants.

Their sample of 46 subjects included 29 males and 17 females; and the distribution of education levels was also rather skewed:

So how did Oxley et al. measure the subjects' political attitudes about "protective policies"?

The dependent variable is an additive index of 18 issue items based on a standard Wilson-Patterson battery. For each issue, respondents were asked to please indicate whether you agree or disagree with the topic listed, and were given agree, disagree, and uncertain response options. These options were coded as 1 when they indicated support for protective policies (e.g. for agreeing with the death penalty, or for disagreeing with pacifism), 0 when they indicated opposition to protective policies (e.g. for disagreeing with the Patriot act, or agreeing with pornography), and .5 for an uncertain response.

As I understand it, they made no special efforts to create the bimodal distribution of summed opinion scores that resulted:

They then ran two psychophysiological tests on the same set of subjects. One was a measure of "skin conductance level" (SCL) recorded while

[e]ach participant was shown three separate threatening images (a very large spider on the face of a frightened person, a dazed individual with a bloody face, and an open wound with maggots in it) interspersed among a sequence of 33 images.

"Conductance" here is the inverse of resistance, measured in log -Siemens:

After logging the data to normalize the distribution, we computed the change in the mean level of skin conductance (SCL) from the previous interstimulus interval (10 s) to the stimulus of interest (20 s). This calculation isolates the change in skin conductance induced by the stimulus and reduces the effects of baseline variations across participants.

(Issues of units, normalization, and measurement intervals aside, this is basically the same phenomenona that goes under the older and somewhat more familiar name of "Galvanic Skin Response" or GSR, and is caused by changes in the autonomically-induced activity levels of sweat glands.)

The other physiological measure was the EMG associated with startle blinks, elicited in response to unexpected noise bursts, where "higher blink amplitudes are indicative of a heightened 'fear state'".

Here's the authors' graphical presentation of the basic result for the SCL measurements. They divided their 43 subjects into those who scored above the median on the "support for protective policies" questionnaire, and those who scored below the median. There was little difference between the groups in response to the non-threatening pictures, but a statistically-significant difference in response to the threatening pictures:

Nicholas Kristof describes the analogous results in the sex-disgust paper this way:

Liberals released only slightly more moisture in reaction to disgusting images than to photos of fruit. But conservatives' glands went into overdrive.

"Overdrive" is a bit of a stretch, I think for comparison, here's a table from Rosalind Picard and Jennifer Healey, "Affective Wearables", ISWC97, showing SCL changes for five subjects as a result of some simple activities such as standing up or coughing:

The difference in means between Oxley et al.'s "high" and "low" protective-policies groups is about .0125 log S. Assuming that they used natural logs, this difference seems to be about two orders of magnitude less than the (unlogged) amounts that Picard and Healey measured as the effect of coughing log(2.4) = 0.88; log(9.9) = 2.29. (This seems like such a preposterous difference that I wonder whether I've got something seriously wrong here but in any case, I hereby register some skepticism about the "going into overdrive" description)

Without making any cross-study assumptions, we can evaluate the effect size in Oxley et al.'s results. From the graph reproduced above, you can see that the difference between the mean (SCL response to threatening pictures) of the "high" and "low" protective-policies groups is about .0125 log S, with a standard error of about .005 log S. Given that there were 23 subjects in each category, this translates to a standard deviation of about .005*sqrt(23) .024 log S, so that the effect size is about d = .0125/.024 0.52.

Though statistically significant, this is only a moderate-sized effect the associated overlap between the two distributions can be represented graphically this way (assuming normal distributions with equal variances):

We can quantify this difference in intuitive terms by calculating how often a randomly-selected person from the "high support for protective policies" group would have a greater average skin-conductance change, on viewing threatening pictures, than a randomly-selected person from the "low support for protective policies" group would: about 64% of the time.

We can also compare another effect from the same study, namely amount of education. This factor is presumably a matter of "individuals' experiences" rather than biology but contrary to what (Kristof's description of) this line of research might lead to you to think, its predictive effect in this experiments was as large as or larger than the effect of the "biological" variables.

As a result of fitting this regression equation

they derived the following coefficients:

The coefficient of -1.63 for the six-level Education scale means that someone in the "College-Plus" categories was predicted to be 5*1.63 = 8.15 (out of 18) points lower on the "protective policies" scale than someone with a high school education (other things in the regression being equal).

What about the magnitude of the skin-conductance effect? Well, the coefficient is 92.2, which means that the more a subject's SCL reacted to threatening images, the higher their "protective policies" attitude scale was. But recall that the average SCL measurement for the high-protective group was about 0.01, and the average SCL measurement for the low-protective group was around -0.0025 (these measures are log -Siemens, so a negative value just means that the conductance measurement was below 1). So the predicted average effect is around 0.01*92.2= 0.922 for the high-protective group, and -0.0025*92.2 = -0.2305 for the "low-protective" group, or an overall mean difference of about 1.15 (out of 18) between the groups attributable to the skin-conductance differences.

In the Supplementary Online Material, we learn the the overall standard deviation of their SCL data was 0.022 log -Siemens. So to equal the predicted effect of 8.15 political-attitude points between high-school and college-plus, we'd need an SCR difference of 8.15/(92.2*0.022) = 4.02 standard deviations.

We can draw a similar conclusion by comparing the standardized regression coefficients of 0.377 for SCL and -0.42 for education.

So the SCL "effect size" was modest, not only in the sense that the difference in means between two politically-defined groups was only about half of the within-group standard deviation, but also in the sense that it appears to have somewhat less predictive leverage than educational level does.

The same comparisons for eye-blink amplitudes lead to similar conclusions. The easiest way to see this is by comparing the standardized regression coefficients for Mean (eye-blink) amplitudes (0.286) and Education (-0.458):

For this model, the predicted effect of the span from highschool to college-plus is 5*-1.76 = -8.8 political-attitude units, and (since the SOM gives the standard deviation of eye-blink amplitude as 0.93) we'd need 8.8/(0.93*1.67) = 5.7 standard deviations of change in eye-blink amplitude to have the same effect as the educational span.

All of this suggests that the "biological" factors of SCL and eye-blink amplitude are somewhat less potent, in this experiment, than the "social" factor of educational level. But it seems to me that in this case, the "biological" factors might turn out to be partly "social" factors after all.

Why? Well, it's likely that political opinion in Lincoln divides to some extent along town-gown lines. It's also plausible that town-gown uneasiness is not unknown in Lincoln, and that a non-academic with conservative opinions, coming into a University laboratory to get wired up with electrodes for a study of political opinions, might have a somewhat higher overall anxiety level than university faculty, students or staff would. This could be true independent of educational level my own experience is that non-academic lawyers and doctors, for example, often seem a bit uneasy around academics. (And vice versa, but in this case it's the academics who are wielding the electrodes.)

Furthermore, people who are more on edge, in an unfamiliar environment that they perceive as potentially hostile or at least potentially embarrassing, are likely to show greater autonomic reactions to negative stimuli. (In a crude analogy, suspenseful music and unsettling camera angles tend to make you jump higher when the cinematic equivalent of "threatening images" appear.)

An argument of this form suggests that (at least some fraction of) the (already modest) statistical effect of the physiological variables on political opinion might not reflect individual "dispositions" at all (whether innate or acquired), but rather might be the result of greater social distance (from the experimental setting) for a significant fraction of the people at one end of the political spectrum being tested.

Despite my skeptical comments, it seems to me that this work is interesting and worthwhile. It's entirely plausible that political attitudes are associated with "dispositional temperaments", and it's also plausible that "dispositional temperaments" owe something to nature as well as nurture. But it seems to me that it's exactly when theories are plausible that we should be most scrupulous in examining the evidence: as Dick Hamming used to say, "Beware of finding what you're looking for".



political protective effect policies scl ]]>Language Log
First shared by - christomer
syndication+ 0 | Search 1 | Shares 1

Carl Zimmer, among other readers, has pointed me to the latest outbreak of bio-political punditry. This time it isn't David Brooks, but rather Nicholas Kristof, "Our Politics May Be All in Our Head", 2/13/2010:

We all know that liberals and conservatives are far apart on health care. But in the way their brains work? Even in automatic reflexes, like blinking? Or the way their glands secrete moisture?

That's the suggestion of some recent research. It hints that the roots of political judgments may lie partly in fundamental personality types and even in the hard-wiring of our brains.

Researchers have found, for example, that some humans are particularly alert to threats, particularly primed to feel vulnerable and perceive danger. Those people are more likely to be conservatives.

Kristof's "recent research" link goes to Kevin Smith, Douglas Oxley, Matthew Hibbing, John Alford, and John Hibbing, "The Ick Factor: Disgust Sensitivity as a Predictor of Political Attitudes". The abstract:

Mounting evidence suggests political attitudes connect to broad, dispositional, perhaps biological temperaments. We add to these empirical results by reporting a correlation between certain political attitudes and physiological reactions to disgusting stimuli such as human excrement and worms being eaten. Specifically, we find that individuals whose skin conductance levels increase when viewing disgusting images are more likely to adopt conservative positions on homosexuality and perhaps a broader range of sex and reproduction issues, a result that serves as a useful illustration of the larger point that political attitudes cannot be separated from generic physiological traits.

This is listed as a "Paper prepared for presentation at the annual meeting of the Midwest Political Science Association", April 2008; but unfortunately, the copy available at the link given is almost impossible to read (because the type is so small, with no method that I could find to enlarge it), and I couldn't find another copy on line. However, a similar set of ideas and experiments are presented in a more accessible way in Douglas Oxley, et al., "Political Attitudes Vary with Physiological Traits", Science 321:5896, 9/19/2008:

Although political views have been thought to arise largely from individuals' experiences, recent research suggests that they may have a biological basis. We present evidence that variations in political attitudes correlate with physiological traits. In a group of 46 adult participants with strong political beliefs, individuals with measurably lower physical sensitivities to sudden noises and threatening visual images were more likely to support foreign aid, liberal immigration policies, pacifism, and gun control, whereas individuals displaying measurably higher physiological reactions to those same stimuli were more likely to favor defense spending, capital punishment, patriotism, and the Iraq War. Thus, the degree to which individuals are physiologically responsive to threat appears to indicate the degree to which they advocate policies that protect the existing social structure from both external (outgroup) and internal (norm-violator) threats.

In other words, the Science paper deals with reactions to sudden noises and theatening visual images, correlated with political attitudes toward level of support for what the authors call "protective policies". The nearly-unreadable conference paper makes an analogous argument about reactions to disgusting images correlated with political attitudes towards "sex and reproduction issues", especially homosexuality. So I'm going to discuss the Science paper if someone can point me to a readable copy of the "disgust" paper, I'll be happy to apply the same analysis there.

(By the way, you can see from the quoted abstracts that Kristof's account of this work is not very different from what the authors themselves have to say about it. This is not a case of egregious journalistic misunderstanding or over-interpretation.)

In the rest of this post, I'm going to go over some aspects of the analysis in the Science paper, and argue that the explanation offered is at least excessive and possibly wrong. I'm somewhat hampered in doing this by the fact that some relevant aspects of the authors' data are not published, and also by (what I found to be) some ambiguities in the presentation of their data and their models. If as a result I've misunderstand what they did or how they interpreted it, I apologize in advance. I plan to write to the authors and ask for clarification and for a copy of their original data, which would allow me (and others) to make a more full and fair evaluation.

First, who were the subjects?

Subjects were recruited in May of 2007 by the Bureau of Sociological Research at the University of Nebraska-Lincoln (BOSR). BOSR contacted a random sample of residents of Lincoln, Nebraska. This initial telephone call followed an introductory letter and was used to pose a limited number of items to respondents with the intention of obtaining a group of individuals with strong political convictions toward whom intense and more focused investigation would be directed. The following three questions were used to screen potential subjects. Yes or no response categories were given for all three questions.

1. Do you follow politics or political issues closely?

2. Is there a certain political issue or set of political issues you feel strongly about?

3. Have you ever supported a particular political issue or cause?

A total of 1310 people were contacted and 608 of them completed the screening items. Subjects were recruited for this particular project only if they responded yes to all three questions. A total of 143 respondents did so and were agreeable in principle to coming to the lab. BOSR was able to schedule (and secure the attendance of) 48 individuals at both sessions (survey and physiological) of this project. Health problems rendered the data from one individual unusable and mechanical problems with a sensor spoiled the data from another, leaving a final total of 46 participants.

Their sample of 46 subjects included 29 males and 17 females; and the distribution of education levels was also rather skewed:

So how did Oxley et al. measure the subjects' political attitudes about "protective policies"?

The dependent variable is an additive index of 18 issue items based on a standard Wilson-Patterson battery. For each issue, respondents were asked to please indicate whether you agree or disagree with the topic listed, and were given agree, disagree, and uncertain response options. These options were coded as 1 when they indicated support for protective policies (e.g. for agreeing with the death penalty, or for disagreeing with pacifism), 0 when they indicated opposition to protective policies (e.g. for disagreeing with the Patriot act, or agreeing with pornography), and .5 for an uncertain response.

As I understand it, they made no special efforts to create the bimodal distribution of summed opinion scores that resulted:

They then ran two psychophysiological tests on the same set of subjects. One was a measure of "skin conductance level" (SCL) recorded while

[e]ach participant was shown three separate threatening images (a very large spider on the face of a frightened person, a dazed individual with a bloody face, and an open wound with maggots in it) interspersed among a sequence of 33 images.

"Conductance" here is the inverse of resistance, measured in log -Siemens:

After logging the data to normalize the distribution, we computed the change in the mean level of skin conductance (SCL) from the previous interstimulus interval (10 s) to the stimulus of interest (20 s). This calculation isolates the change in skin conductance induced by the stimulus and reduces the effects of baseline variations across participants.

(Issues of units, normalization, and measurement intervals aside, this is basically the same phenomenona that goes under the older and somewhat more familiar name of "Galvanic Skin Response" or GSR, and is caused by changes in the autonomically-induced activity levels of sweat glands.)

The other physiological measure was the EMG associated with startle blinks, elicited in response to unexpected noise bursts, where "higher blink amplitudes are indicative of a heightened 'fear state'".

Here's the authors' graphical presentation of the basic result for the SCL measurements. They divided their 43 subjects into those who scored above the median on the "support for protective policies" questionnaire, and those who scored below the median. There was little difference between the groups in response to the non-threatening pictures, but a statistically-significant difference in response to the threatening pictures:

Nicholas Kristof describes the analogous results in the sex-disgust paper this way:

Liberals released only slightly more moisture in reaction to disgusting images than to photos of fruit. But conservatives' glands went into overdrive.

"Overdrive" is a bit of a stretch, I think for comparison, here's a table from Rosalind Picard and Jennifer Healey, "Affective Wearables", ISWC97, showing SCL changes for five subjects as a result of some simple activities such as standing up or coughing:

The difference in means between Oxley et al.'s "high" and "low" protective-policies groups is about .0125 log S. Assuming that they used natural logs, this difference seems to be about two orders of magnitude less than the (unlogged) amounts that Picard and Healey measured as the effect of coughing log(2.4) = 0.88; log(9.9) = 2.29. (This seems like such a preposterous difference that I wonder whether I've got something seriously wrong here but in any case, I hereby register some skepticism about the "going into overdrive" description)

Without making any cross-study assumptions, we can evaluate the effect size in Oxley et al.'s results. From the graph reproduced above, you can see that the difference between the mean (SCL response to threatening pictures) of the "high" and "low" protective-policies groups is about .0125 log S, with a standard error of about .005 log S. Given that there were 23 subjects in each category, this translates to a standard deviation of about .005*sqrt(23) .024 log S, so that the effect size is about d = .0125/.024 0.52.

Though statistically significant, this is only a moderate-sized effect the associated overlap between the two distributions can be represented graphically this way (assuming normal distributions with equal variances):

We can quantify this difference in intuitive terms by calculating how often a randomly-selected person from the "high support for protective policies" group would have a greater average skin-conductance change, on viewing threatening pictures, than a randomly-selected person from the "low support for protective policies" group would: about 64% of the time.

We can also compare another effect from the same study, namely amount of education. This factor is presumably a matter of "individuals' experiences" rather than biology but contrary to what (Kristof's description of) this line of research might lead to you to think, its predictive effect in this experiments was as large as or larger than the effect of the "biological" variables.

As a result of fitting this regression equation

they derived the following coefficients:

The coefficient of -1.63 for the six-level Education scale means that someone in the "College-Plus" categories was predicted to be 5*1.63 = 8.15 (out of 18) points lower on the "protective policies" scale than someone with a high school education (other things in the regression being equal).

What about the magnitude of the skin-conductance effect? Well, the coefficient is 92.2, which means that the more a subject's SCL reacted to threatening images, the higher their "protective policies" attitude scale was. But recall that the average SCL measurement for the high-protective group was about 0.01, and the average SCL measurement for the low-protective group was around -0.0025 (these measures are log -Siemens, so a negative value just means that the conductance measurement was below 1). So the predicted average effect is around 0.01*92.2= 0.922 for the high-protective group, and -0.0025*92.2 = -0.2305 for the "low-protective" group, or an overall mean difference of about 1.15 (out of 18) between the groups attributable to the skin-conductance differences.

In the Supplementary Online Material, we learn the the overall standard deviation of their SCL data was 0.022 log -Siemens. So to equal the predicted effect of 8.15 political-attitude points between high-school and college-plus, we'd need an SCR difference of 8.15/(92.2*0.022) = 4.02 standard deviations.

We can draw a similar conclusion by comparing the standardized regression coefficients of 0.377 for SCL and -0.42 for education.

So the SCL "effect size" was modest, not only in the sense that the difference in means between two politically-defined groups was only about half of the within-group standard deviation, but also in the sense that it appears to have somewhat less predictive leverage than educational level does.

The same comparisons for eye-blink amplitudes lead to similar conclusions. The easiest way to see this is by comparing the standardized regression coefficients for Mean (eye-blink) amplitudes (0.286) and Education (-0.458):

For this model, the predicted effect of the span from highschool to college-plus is 5*-1.76 = -8.8 political-attitude units, and (since the SOM gives the standard deviation of eye-blink amplitude as 0.93) we'd need 8.8/(0.93*1.67) = 5.7 standard deviations of change in eye-blink amplitude to have the same effect as the educational span.

All of this suggests that the "biological" factors of SCL and eye-blink amplitude are somewhat less potent, in this experiment, than the "social" factor of educational level. But it seems to me that in this case, the "biological" factors might turn out to be partly "social" factors after all.

Why? Well, it's likely that political opinion in Lincoln divides to some extent along town-gown lines. It's also plausible that town-gown uneasiness is not unknown in Lincoln, and that a non-academic with conservative opinions, coming into a University laboratory to get wired up with electrodes for a study of political opinions, might have a somewhat higher overall anxiety level than university faculty, students or staff would. This could be true independent of educational level my own experience is that non-academic lawyers and doctors, for example, often seem a bit uneasy around academics. (And vice versa, but in this case it's the academics who are wielding the electrodes.)

Furthermore, people who are more on edge, in an unfamiliar environment that they perceive as potentially hostile or at least potentially embarrassing, are likely to show greater autonomic reactions to negative stimuli. (In a crude analogy, suspenseful music and unsettling camera angles tend to make you jump higher when the cinematic equivalent of "threatening images" appear.)

An argument of this form suggests that (at least some fraction of) the (already modest) statistical effect of the physiological variables on political opinion might not reflect individual "dispositions" at all (whether innate or acquired), but rather might be the result of greater social distance (from the experimental setting) for a significant fraction of the people at one end of the political spectrum being tested.

Despite my skeptical comments, it seems to me that this work is interesting and worthwhile. It's entirely plausible that political attitudes are associated with "dispositional temperaments", and it's also plausible that "dispositional temperaments" owe something to nature as well as nurture. But it seems to me that it's exactly when theories are plausible that we should be most scrupulous in examining the evidence: as Dick Hamming used to say, "Beware of finding what you're looking for".



political protective effect policies scl ]]> Tue, 16 Feb 2010 07:45:57 -0500 30:00 tag:filome.com,7 UI Test Automation Tools are Snake Oil http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil Shared item Object Mentor Blog
First shared by - robdiana
syndication+ 0 | Search 1 | Shares 1

It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn't matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say we tried. I say, don't worry, everyone does.

It's a very familiar rabbit hole' in the industry. It's sort of like the old days when you'd find a couple of classes generated by a CASE tool in every code base you visited if you looked hard enough. People started with the merry notion that they were going to round-trip code with some CASE tool and they learned (like most lucky teams do) that it just doesn't pay for itself, it's not worth the time or the frustration. UI Test Automation tools are in the same category. Personally, I think that in this day and age selling them is irresponsible. Developing them open-source? Well, let your conscience be your guide, but really, even though people can use them responsibly, they hardly ever do because these tools are sold with a dream, a very seductive and dangerous one.

The Dream

Janet comes into work in the morning and she sits down at her super-duper testing console. She presses a button and the testing system springs to life. The application comes up all at once across ten monitors. Cursors move, selections are made (silently) and tests run against the user interface magically, as if some eager set of ghost elves took control, mischievously burrowing through the nooks and crannies of the application, running scripts to completion, and making little notes whenever there is a failure. Janet sits back in her chair, waiting for the elves to report back to her. She stirs her coffee gently.

The Reality

Janet hasn't gone home yet. It's 2AM and she has to report completion of all her test cases at a meeting in the morning. She thinks she's past the last configuration issue but she's not sure. For the last hour, she's been trying to make sure that a particular button is pressed at step 14 of her script, but quirky latency on the server is preventing it from happening consistently. Sadly, she has to run the script from the beginning each time. Oh, and five hours ago she discovered UI changes which invalidated 30% of the regression tests. Most of the changes were easy but she still has 12 cases to go and her 9AM meeting looms ahead of her.

This gap between the dream and the reality is not a matter of flawed execution, it's endemic. Here's the scoop.

UI Test Automation Tools are Brittle

You might not think this is fair but it is, really. I haven't seen one of these tools yet which isn't susceptible to missed events or timing madness. It just happens. The fact of the matter is, it is hard to sit on the outside of an application an instrument it. It's a very technology sensitive problem. You need to hook into either the OS or the browser or both. Neither are ever really built from the ground up for that sort of access.

UI Based Testing Is Not the Solution That Vendors Imply It Is

This is the big issue, the one which really hurts the industry. The fact of the matter is that UI based testing should be used for UIs: that's it. You should not be testing your full application end-to-end through a UI Testing tool. First of all, that sort of testing couples some of the most important tests in your system to one of the most volatile parts of it. It's easy for us to see business logic as volatile, but really, the UI is the thing which twists and ripples in the winds of change. When customers want new features, often those features involve new workflows. When usability experts discover better ways of models of interaction, an agile business seizes upon them and makes the changesif they can. You'd be surprised at the number of applications which continue to sport out of day user interfaces simply because the development organization is terrified of throwing away all of their regression tests which (by the way) go through the UI. Even if you're not a consultant like me, visiting teams and seeing their development processes, you can see hints from the outside. Think of every website or shrink-wrap application which has the same old workflow and a UI that has become more cluttered over the years. Often it's because of that lock-in.

UI Based Testing Takes More Staff and Time Than You Expect

This, really, is the most common failure case. It's the case which explains the dust on the testing box's keyboard. Someone, usually disconnected from the development organization, decides that hey, we need to solve the testing problem. We have too many people doing manual testing. It's taking forever. So, they do their research, find a vendor with with a good licensing model and a good pitch and then they push it on the development organization. They are, of course, looking to reduce staff so when they realize that translating all of those automated tests to the tool takes quite a while, they are taken aback. But, of course, it is just a temporary cost, right? But, it takes far longer than they expect. Remember Janet's story? It's really hard to catch up with a UI-Based testing tool. It's hard to even stay in place with one. Typically it takes a number of people to do UI-Based automated testing for a development team in sync with an iteration and worse, they'll always lag behind a bit because you can't really write UI-based tests ahead of time the way you can with FIT and other beneath-the-UI testing tools. From what I've seen UI-based testing, done diligently, takes the effort of about one tester for every two to three developers. That's what it seems to cost amortized across all of the maintenance of UI-induced test breakage. Oh, and by the way, if think you are going to save labor using record and playback? Nope, you aren't. It doesn't work.

Solutions

The fact of the matter is, you can use these tools effectively, but in a very narrow space. It's nice to be able to test the UIby itself. However, this sort of thing requires an architectural change ( http://www.computer.org/portal/web/csdl/doi/10.1109/MS.2005.110 ). In general, UIs are too volatile for end-to-end testing. Teams that do it well, typically develop a small task-focused scripting layer and build tests on top if it so that the actual tests don't touch the UI directly. But, if they happen across that technique, they are lucky. Still, it isn't an ideal solution. You really want to be below the UI working against an API which exposes the business logic. And, because of that nearly mystical synergy between testability and good design, that API layer is often useful for many things other than testing.

Conclusion/Challenge

I recognize that I've been rather vicious in the this blog. If you develop these tools for a living, you might not think it's fair. But consider this. If you don't think I'm being fair, take a look at how your tools are marketed. In particular, show me where the product literature discourages end-to-end testing through the tool. Otherwise, well, you know, you are probably developing snake-oil.



testing ui tools tests end ]]>Object Mentor Blog
First shared by - robdiana
syndication+ 0 | Search 1 | Shares 1

It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn't matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say we tried. I say, don't worry, everyone does.

It's a very familiar rabbit hole' in the industry. It's sort of like the old days when you'd find a couple of classes generated by a CASE tool in every code base you visited if you looked hard enough. People started with the merry notion that they were going to round-trip code with some CASE tool and they learned (like most lucky teams do) that it just doesn't pay for itself, it's not worth the time or the frustration. UI Test Automation tools are in the same category. Personally, I think that in this day and age selling them is irresponsible. Developing them open-source? Well, let your conscience be your guide, but really, even though people can use them responsibly, they hardly ever do because these tools are sold with a dream, a very seductive and dangerous one.

The Dream

Janet comes into work in the morning and she sits down at her super-duper testing console. She presses a button and the testing system springs to life. The application comes up all at once across ten monitors. Cursors move, selections are made (silently) and tests run against the user interface magically, as if some eager set of ghost elves took control, mischievously burrowing through the nooks and crannies of the application, running scripts to completion, and making little notes whenever there is a failure. Janet sits back in her chair, waiting for the elves to report back to her. She stirs her coffee gently.

The Reality

Janet hasn't gone home yet. It's 2AM and she has to report completion of all her test cases at a meeting in the morning. She thinks she's past the last configuration issue but she's not sure. For the last hour, she's been trying to make sure that a particular button is pressed at step 14 of her script, but quirky latency on the server is preventing it from happening consistently. Sadly, she has to run the script from the beginning each time. Oh, and five hours ago she discovered UI changes which invalidated 30% of the regression tests. Most of the changes were easy but she still has 12 cases to go and her 9AM meeting looms ahead of her.

This gap between the dream and the reality is not a matter of flawed execution, it's endemic. Here's the scoop.

UI Test Automation Tools are Brittle

You might not think this is fair but it is, really. I haven't seen one of these tools yet which isn't susceptible to missed events or timing madness. It just happens. The fact of the matter is, it is hard to sit on the outside of an application an instrument it. It's a very technology sensitive problem. You need to hook into either the OS or the browser or both. Neither are ever really built from the ground up for that sort of access.

UI Based Testing Is Not the Solution That Vendors Imply It Is

This is the big issue, the one which really hurts the industry. The fact of the matter is that UI based testing should be used for UIs: that's it. You should not be testing your full application end-to-end through a UI Testing tool. First of all, that sort of testing couples some of the most important tests in your system to one of the most volatile parts of it. It's easy for us to see business logic as volatile, but really, the UI is the thing which twists and ripples in the winds of change. When customers want new features, often those features involve new workflows. When usability experts discover better ways of models of interaction, an agile business seizes upon them and makes the changesif they can. You'd be surprised at the number of applications which continue to sport out of day user interfaces simply because the development organization is terrified of throwing away all of their regression tests which (by the way) go through the UI. Even if you're not a consultant like me, visiting teams and seeing their development processes, you can see hints from the outside. Think of every website or shrink-wrap application which has the same old workflow and a UI that has become more cluttered over the years. Often it's because of that lock-in.

UI Based Testing Takes More Staff and Time Than You Expect

This, really, is the most common failure case. It's the case which explains the dust on the testing box's keyboard. Someone, usually disconnected from the development organization, decides that hey, we need to solve the testing problem. We have too many people doing manual testing. It's taking forever. So, they do their research, find a vendor with with a good licensing model and a good pitch and then they push it on the development organization. They are, of course, looking to reduce staff so when they realize that translating all of those automated tests to the tool takes quite a while, they are taken aback. But, of course, it is just a temporary cost, right? But, it takes far longer than they expect. Remember Janet's story? It's really hard to catch up with a UI-Based testing tool. It's hard to even stay in place with one. Typically it takes a number of people to do UI-Based automated testing for a development team in sync with an iteration and worse, they'll always lag behind a bit because you can't really write UI-based tests ahead of time the way you can with FIT and other beneath-the-UI testing tools. From what I've seen UI-based testing, done diligently, takes the effort of about one tester for every two to three developers. That's what it seems to cost amortized across all of the maintenance of UI-induced test breakage. Oh, and by the way, if think you are going to save labor using record and playback? Nope, you aren't. It doesn't work.

Solutions

The fact of the matter is, you can use these tools effectively, but in a very narrow space. It's nice to be able to test the UIby itself. However, this sort of thing requires an architectural change ( http://www.computer.org/portal/web/csdl/doi/10.1109/MS.2005.110 ). In general, UIs are too volatile for end-to-end testing. Teams that do it well, typically develop a small task-focused scripting layer and build tests on top if it so that the actual tests don't touch the UI directly. But, if they happen across that technique, they are lucky. Still, it isn't an ideal solution. You really want to be below the UI working against an API which exposes the business logic. And, because of that nearly mystical synergy between testability and good design, that API layer is often useful for many things other than testing.

Conclusion/Challenge

I recognize that I've been rather vicious in the this blog. If you develop these tools for a living, you might not think it's fair. But consider this. If you don't think I'm being fair, take a look at how your tools are marketed. In particular, show me where the product literature discourages end-to-end testing through the tool. Otherwise, well, you know, you are probably developing snake-oil.



testing ui tools tests end ]]> Mon, 04 Jan 2010 07:42:02 -0500 30:00 tag:filome.com,8 If You Don't Have Test Automation, Don't Bother http://feedproxy.google.com/~r/EricJacobsonSoftwareTestingBlog/~3/n0ODWa4p3xg/if-you-dont-have-test-automation-dont.html Shared item Test This Blog - Eric Jacobson's Software Testing Blog
First shared by - Mariela
syndication+ 12 | Search 1 | Shares 1

Shared by Mariela
"We spend more time trying to get the darn thing to run unattended and stop flagging false bugs ..." - I understand his pain
Successful test automation is the elephant in the room for many testers. We all want to do it because manual testing is hard, our manager and devs would think we were bad-ass, andoh yeah, some of us believe it would improve our AUT quality. We fantasize about triggering our automated test stack and going home, while the manual testers toil away. We would even let them kiss the tips of our fingers as we walked out the door.

sounds good.

So we (testers) make an attempt at automation, exaggerate the success, then eventually feel like losers. We spend more time trying to get the darn thing to run unattended and stop flagging false bugs, while the quality of our tests takes a back seat and our available test time shrinks.

We were testing one product. Now we are testing two.

The two obvious problems are, 1.) Most of us are not developers. 2.) Writing a program to test another program is more difficult than writing the original program. ...Ah yes, a match made in heaven!

I watched an automated testing webinar last week. It was more honest than I expected. The claim was, to be successful at test automation the team should not expect existing testers to start automating tests. Instead, a new team of developers should be added to automate tests that testers write. This new team would have their own requirement reviews, manage their own code base, and have their own testers to test their test automation stack. This does not sound cheap!

While watching this webinar, something occurred to me. Maybe we don't need test automation. Why do I think this? Simple. Because somehow my team is managing to release successful software to the company without it. There is no test automation team on our payroll. Regression testing is spotty at best, yet somehow our team is considered a model of success within the company. How is this possible when every other test tool spam email or blog post I read makes some reference to test automation?

In my case, I believe a few things have made this possible:
  • The devs are talented and organized enough to minimize the amount of stuff they break with new builds. This makes regression testing less important for us testers.
  • The BAs are talented enough to understand how new features impact existing features.
  • The testers are talented enough to know where to look. And they work closely with devs and BAs to determine how stuff should work.
  • The user support team is highly accessible to users, knowledgeable about the AUT and the business, and works closely with the BAs/devs/testers to get the right prod bugs patched quickly. The entire team is committed to serving the users.
  • The users are sophisticated enough to communicate bug details and use workarounds when waiting on fixes. The users like us because we make their jobs easier. The users want us to succeed so we can keep making their jobs easier.
  • The possibility of prod bugs resulting in death, loss of customers, or other massive financial loss is slim to none.
I suspect a great deal of software teams are similar to mine. I'm interested in hearing from other software teams that do not depend on tester-driven test automation.

I do use test automation to help me with one of my simple AUTs which happens to lend itself to automated testing (see JART). However, from my experiences, there are few apps that are easy to automate with simple checks.


test automation testers team testing ]]>Test This Blog - Eric Jacobson's Software Testing Blog
First shared by - Mariela
syndication+ 12 | Search 1 | Shares 1

Shared by Mariela
"We spend more time trying to get the darn thing to run unattended and stop flagging false bugs ..." - I understand his pain
Successful test automation is the elephant in the room for many testers. We all want to do it because manual testing is hard, our manager and devs would think we were bad-ass, andoh yeah, some of us believe it would improve our AUT quality. We fantasize about triggering our automated test stack and going home, while the manual testers toil away. We would even let them kiss the tips of our fingers as we walked out the door.

sounds good.

So we (testers) make an attempt at automation, exaggerate the success, then eventually feel like losers. We spend more time trying to get the darn thing to run unattended and stop flagging false bugs, while the quality of our tests takes a back seat and our available test time shrinks.

We were testing one product. Now we are testing two.

The two obvious problems are, 1.) Most of us are not developers. 2.) Writing a program to test another program is more difficult than writing the original program. ...Ah yes, a match made in heaven!

I watched an automated testing webinar last week. It was more honest than I expected. The claim was, to be successful at test automation the team should not expect existing testers to start automating tests. Instead, a new team of developers should be added to automate tests that testers write. This new team would have their own requirement reviews, manage their own code base, and have their own testers to test their test automation stack. This does not sound cheap!

While watching this webinar, something occurred to me. Maybe we don't need test automation. Why do I think this? Simple. Because somehow my team is managing to release successful software to the company without it. There is no test automation team on our payroll. Regression testing is spotty at best, yet somehow our team is considered a model of success within the company. How is this possible when every other test tool spam email or blog post I read makes some reference to test automation?

In my case, I believe a few things have made this possible:
  • The devs are talented and organized enough to minimize the amount of stuff they break with new builds. This makes regression testing less important for us testers.
  • The BAs are talented enough to understand how new features impact existing features.
  • The testers are talented enough to know where to look. And they work closely with devs and BAs to determine how stuff should work.
  • The user support team is highly accessible to users, knowledgeable about the AUT and the business, and works closely with the BAs/devs/testers to get the right prod bugs patched quickly. The entire team is committed to serving the users.
  • The users are sophisticated enough to communicate bug details and use workarounds when waiting on fixes. The users like us because we make their jobs easier. The users want us to succeed so we can keep making their jobs easier.
  • The possibility of prod bugs resulting in death, loss of customers, or other massive financial loss is slim to none.
I suspect a great deal of software teams are similar to mine. I'm interested in hearing from other software teams that do not depend on tester-driven test automation.

I do use test automation to help me with one of my simple AUTs which happens to lend itself to automated testing (see JART). However, from my experiences, there are few apps that are easy to automate with simple checks.


test automation testers team testing ]]> Sat, 24 Oct 2009 05:32:33 -0400 30:00 tag:filome.com,9 Scientific Speed Reading: How to Read 300% Faster in 20 Minutes http://feedproxy.google.com/~r/timferriss/~3/6aHTOlUdorg/ Shared item The Blog of Author Tim Ferriss
First shared by - Jorg
syndication+ 90 | Search 1 | Shares 1


(Photo: Dustin Diaz)

How much more could you get done if you completed all of your required reading in 1/3 or 1/5 the time?

Increasing reading speed is a process of controlling fine motor movementperiod.

This post is a condensed overview of principles I taught to undergraduates at Princeton University in 1998 at a seminar called the PX Project. The below was written several years ago, so it's worded like Ivy-Leaguer pompous-ass prose, but the results are substantial. In fact, while on an airplane in China two weeks ago, I helped Glenn McElhose increase his reading speed 34% in less than 5 minutes.

I have never seen the method fail. Here's how it works

The PX Project

The PX Project, a single 3-hour cognitive experiment, produced an average increase in reading speed of 386%.

It was tested with speakers of five languages, and even dyslexics were conditioned to read technical material at more than 3,000 words-per-minute (wpm), or 10 pages per minute. One page every 6 seconds. By comparison, the average reading speed in the US is 200-300 wpm (1/2 to 1 page per minute), with the top 1% of the population reading over 400 wpm

If you understand several basic principles of the human visual system, you can eliminate inefficiencies and increase speed while improving retention.

To perform the exercises in this post and see the results, you will need: a book of 200+ pages that can lay flat when open, a pen, and a timer (a stop watch with alarm or kitchen timer is ideal). You should complete the 20 minutes of exercises in one session.

First, several definitions and distinctions specific to the reading process:

A) Synopsis: You must minimize the number and duration of fixations per line to increase speed.

You do not read in a straight line, but rather in a sequence of saccadic movements (jumps). Each of these saccades ends with a fixation, or a temporary snapshot of the text within you focus area (approx. the size of a quarter at 8? from reading surface). Each fixation will last to seconds in the untrained subject. To demonstrate this, close one eye, place a fingertip on top of that eyelid, and then slowly scan a straight horizontal line with your other eye-you will feel distinct and separate movements and periods of fixation.

B) Synopsis: You must eliminate regression and back-skipping to increase speed.

The untrained subject engages in regression (conscious rereading) and back-skipping (subconscious rereading via misplacement of fixation) for up to 30% of total reading time.

C) Synopsis: You must use conditioning drills to increase horizontal peripheral vision span and the number of words registered per fixation.

Untrained subjects use central focus but not horizontal peripheral vision span during reading, foregoing up to 50% of their words per fixation (the number of words that can be perceived and read in each fixation).

The Protocol

You will 1) learn technique, 2) learn to apply techniques with speed through conditioning, then 3) learn to test yourself with reading for comprehension.

These are separate, and your adaptation to the sequencing depends on keeping them separate. Do not worry about comprehension if you are learning to apply a motor skill with speed, for example. The adaptive sequence is: technique technique with speed comprehensive reading testing.

As a general rule, you will need to practice technique at 3x the speed of your ultimate target reading speed. Thus, if you currently read at 300 wpm and your target reading speed is 900 wpm, you will need to practice technique at 1,800 words-per-minute, or 6 pages per minute (10 seconds per page).

We will cover two main techniques in this introduction:

1) Trackers and Pacers (to address A and B above)
2) Perceptual Expansion (to address C)

First Determining Baseline

To determine your current reading speed, take your practice book (which should lay flat when open on a table) and count the number of words in 5 lines. Divide this number of words by 5, and you have your average number of words-per-line.

Example: 62 words/5 lines = 12.4, which you round to 12 words-per-line

Next, count the number of text lines on 5 pages and divide by 5 to arrive at the average number of lines per page. Multiply this by average number of words-per-line, and you have your average number of words per page.

Example: 154 lines/5 pages = 30.8, rounded to 31 lines per page x 12 words-per-line = 372 words per page

Mark your first line and read with a timer for 1 minute exactly-do not read faster than normal, and read for comprehension. After exactly one minute, multiply the number of lines by your average words-per-line to determine your current words-per-minute (wpm) rate.

Second Trackers and Pacers

Regression, back-skipping, and the duration of fixations can be minimized by using a tracker and pacer. To illustrate the importance of a tracker-did you use a pen or finger when counting the number of words or lines in above baseline calculations? If you did, it was for the purpose of tracking-using a visual aid to guide fixation efficiency and accuracy. Nowhere is this more relevant than in conditioning reading speed by eliminating such inefficiencies.

For the purposes of this article, we will use a pen. Holding the pen in your dominant hand, you will underline each line (with the cap on), keeping your eye fixation above the tip of the pen. This will not only serve as a tracker, but it will also serve as a pacer for maintaining consistent speed and decreasing fixation duration. You may hold it as you would when writing, but it is recommended that you hold it under your hand, flat against the page.

1) Technique (2 minutes):

Practice using the pen as a tracker and pacer. Underline each line, focusing above the tip of the pen. DO NOT CONCERN YOURSELF WITH COMPREHENSION. Keep each line to a maximum of 1 second, and increase the speed with each subsequent page. Read, but under no circumstances should you take longer than 1 second per line.

2) Speed (3 minutes):

Repeat the technique, keeping each line to no more than second (2 lines for a single one-one-thousand). Some will comprehend nothing, which is to be expected. Maintain speed and technique-you are conditioning your perceptual reflexes, and this is a speed exercise designed to facilitate adaptations in your system. Do not decrease speed. second per line for 3 minutes; focus above the pen and concentrate on technique with speed. Focus on the exercise, and do not daydream.

Third Perceptual Expansion

If you focus on the center of your computer screen (focus relating to the focal area of the fovea in within the eye), you can still perceive and register the sides of the screen. Training peripheral vision to register more effectively can increase reading speed over 300%. Untrained readers use up to of their peripheral field on margins by moving from 1st word to last, spending 25-50% of their time reading margins with no content.

To illustrate, let us take the hypothetical one line: Once upon a time, students enjoyed reading four hours a day. If you were able to begin your reading at time and finish the line at four, you would eliminate 6 of 11 words, more than doubling your reading speed. This concept is easy to implement and combine with the tracking and pacing you've already practiced.

1) Technique (1 minute):

Use the pen to track and pace at a consistent speed of one line per second. Begin 1 word in from the first word of each line, and end 1 word in from the last word.

DO NOT CONCERN YOURSELF WITH COMPREHENSION. Keep each line to a maximum of 1 second, and increase the speed with each subsequent page. Read, but under no circumstances should you take longer than 1 second per line.

2) Technique (1 minute):

Use the pen to track and pace at a consistent speed of one line per second. Begin 2 words in from the first word of each line, and end 2 words in from the last word.

3) Speed (3 minutes):

Begin at least 3 words in from the first word of each line, and end 3 words in from the last word. Repeat the technique, keeping each line to no more than second (2 lines for a single one-one-thousand).

Some will comprehend nothing, which is to be expected. Maintain speed and technique-you are conditioning your perceptual reflexes, and this is a speed exercise designed to facilitate adaptations in your system. Do not decrease speed. second per line for 3 minutes; focus above the pen and concentrate on technique with speed. Focus on the exercise, and do not daydream.

Fourth Calculate New WPM Reading Speed

Mark your first line and read with a timer for 1 minute exactly- Read at your fastest comprehension rate. Multiply the number of lines by your previously determined average words-per-line to get determine your new words-per-minute (wpm) rate.

Congratulations on completing your cursory overview of some of the techniques that can be used to accelerate human cognition (defined as the processing and use of information).

Final recommendations: If used for study, it is recommended that you not read 3 assignments in the time it would take you to read one, but rather, read the same assignment 3 times for exposure and recall improvement, depending on relevancy to testing.

Happy trails, page blazers.

Related and Popular:
How to Tim Ferriss Your Love Life
Tim Ferriss at Princeton (video)



speed per words reading read ]]>The Blog of Author Tim Ferriss
First shared by - Jorg
syndication+ 90 | Search 1 | Shares 1


(Photo: Dustin Diaz)

How much more could you get done if you completed all of your required reading in 1/3 or 1/5 the time?

Increasing reading speed is a process of controlling fine motor movementperiod.

This post is a condensed overview of principles I taught to undergraduates at Princeton University in 1998 at a seminar called the PX Project. The below was written several years ago, so it's worded like Ivy-Leaguer pompous-ass prose, but the results are substantial. In fact, while on an airplane in China two weeks ago, I helped Glenn McElhose increase his reading speed 34% in less than 5 minutes.

I have never seen the method fail. Here's how it works

The PX Project

The PX Project, a single 3-hour cognitive experiment, produced an average increase in reading speed of 386%.

It was tested with speakers of five languages, and even dyslexics were conditioned to read technical material at more than 3,000 words-per-minute (wpm), or 10 pages per minute. One page every 6 seconds. By comparison, the average reading speed in the US is 200-300 wpm (1/2 to 1 page per minute), with the top 1% of the population reading over 400 wpm

If you understand several basic principles of the human visual system, you can eliminate inefficiencies and increase speed while improving retention.

To perform the exercises in this post and see the results, you will need: a book of 200+ pages that can lay flat when open, a pen, and a timer (a stop watch with alarm or kitchen timer is ideal). You should complete the 20 minutes of exercises in one session.

First, several definitions and distinctions specific to the reading process:

A) Synopsis: You must minimize the number and duration of fixations per line to increase speed.

You do not read in a straight line, but rather in a sequence of saccadic movements (jumps). Each of these saccades ends with a fixation, or a temporary snapshot of the text within you focus area (approx. the size of a quarter at 8? from reading surface). Each fixation will last to seconds in the untrained subject. To demonstrate this, close one eye, place a fingertip on top of that eyelid, and then slowly scan a straight horizontal line with your other eye-you will feel distinct and separate movements and periods of fixation.

B) Synopsis: You must eliminate regression and back-skipping to increase speed.

The untrained subject engages in regression (conscious rereading) and back-skipping (subconscious rereading via misplacement of fixation) for up to 30% of total reading time.

C) Synopsis: You must use conditioning drills to increase horizontal peripheral vision span and the number of words registered per fixation.

Untrained subjects use central focus but not horizontal peripheral vision span during reading, foregoing up to 50% of their words per fixation (the number of words that can be perceived and read in each fixation).

The Protocol

You will 1) learn technique, 2) learn to apply techniques with speed through conditioning, then 3) learn to test yourself with reading for comprehension.

These are separate, and your adaptation to the sequencing depends on keeping them separate. Do not worry about comprehension if you are learning to apply a motor skill with speed, for example. The adaptive sequence is: technique technique with speed comprehensive reading testing.

As a general rule, you will need to practice technique at 3x the speed of your ultimate target reading speed. Thus, if you currently read at 300 wpm and your target reading speed is 900 wpm, you will need to practice technique at 1,800 words-per-minute, or 6 pages per minute (10 seconds per page).

We will cover two main techniques in this introduction:

1) Trackers and Pacers (to address A and B above)
2) Perceptual Expansion (to address C)

First Determining Baseline

To determine your current reading speed, take your practice book (which should lay flat when open on a table) and count the number of words in 5 lines. Divide this number of words by 5, and you have your average number of words-per-line.

Example: 62 words/5 lines = 12.4, which you round to 12 words-per-line

Next, count the number of text lines on 5 pages and divide by 5 to arrive at the average number of lines per page. Multiply this by average number of words-per-line, and you have your average number of words per page.

Example: 154 lines/5 pages = 30.8, rounded to 31 lines per page x 12 words-per-line = 372 words per page

Mark your first line and read with a timer for 1 minute exactly-do not read faster than normal, and read for comprehension. After exactly one minute, multiply the number of lines by your average words-per-line to determine your current words-per-minute (wpm) rate.

Second Trackers and Pacers

Regression, back-skipping, and the duration of fixations can be minimized by using a tracker and pacer. To illustrate the importance of a tracker-did you use a pen or finger when counting the number of words or lines in above baseline calculations? If you did, it was for the purpose of tracking-using a visual aid to guide fixation efficiency and accuracy. Nowhere is this more relevant than in conditioning reading speed by eliminating such inefficiencies.

For the purposes of this article, we will use a pen. Holding the pen in your dominant hand, you will underline each line (with the cap on), keeping your eye fixation above the tip of the pen. This will not only serve as a tracker, but it will also serve as a pacer for maintaining consistent speed and decreasing fixation duration. You may hold it as you would when writing, but it is recommended that you hold it under your hand, flat against the page.

1) Technique (2 minutes):

Practice using the pen as a tracker and pacer. Underline each line, focusing above the tip of the pen. DO NOT CONCERN YOURSELF WITH COMPREHENSION. Keep each line to a maximum of 1 second, and increase the speed with each subsequent page. Read, but under no circumstances should you take longer than 1 second per line.

2) Speed (3 minutes):

Repeat the technique, keeping each line to no more than second (2 lines for a single one-one-thousand). Some will comprehend nothing, which is to be expected. Maintain speed and technique-you are conditioning your perceptual reflexes, and this is a speed exercise designed to facilitate adaptations in your system. Do not decrease speed. second per line for 3 minutes; focus above the pen and concentrate on technique with speed. Focus on the exercise, and do not daydream.

Third Perceptual Expansion

If you focus on the center of your computer screen (focus relating to the focal area of the fovea in within the eye), you can still perceive and register the sides of the screen. Training peripheral vision to register more effectively can increase reading speed over 300%. Untrained readers use up to of their peripheral field on margins by moving from 1st word to last, spending 25-50% of their time reading margins with no content.

To illustrate, let us take the hypothetical one line: Once upon a time, students enjoyed reading four hours a day. If you were able to begin your reading at time and finish the line at four, you would eliminate 6 of 11 words, more than doubling your reading speed. This concept is easy to implement and combine with the tracking and pacing you've already practiced.

1) Technique (1 minute):

Use the pen to track and pace at a consistent speed of one line per second. Begin 1 word in from the first word of each line, and end 1 word in from the last word.

DO NOT CONCERN YOURSELF WITH COMPREHENSION. Keep each line to a maximum of 1 second, and increase the speed with each subsequent page. Read, but under no circumstances should you take longer than 1 second per line.

2) Technique (1 minute):

Use the pen to track and pace at a consistent speed of one line per second. Begin 2 words in from the first word of each line, and end 2 words in from the last word.

3) Speed (3 minutes):

Begin at least 3 words in from the first word of each line, and end 3 words in from the last word. Repeat the technique, keeping each line to no more than second (2 lines for a single one-one-thousand).

Some will comprehend nothing, which is to be expected. Maintain speed and technique-you are conditioning your perceptual reflexes, and this is a speed exercise designed to facilitate adaptations in your system. Do not decrease speed. second per line for 3 minutes; focus above the pen and concentrate on technique with speed. Focus on the exercise, and do not daydream.

Fourth Calculate New WPM Reading Speed

Mark your first line and read with a timer for 1 minute exactly- Read at your fastest comprehension rate. Multiply the number of lines by your previously determined average words-per-line to get determine your new words-per-minute (wpm) rate.

Congratulations on completing your cursory overview of some of the techniques that can be used to accelerate human cognition (defined as the processing and use of information).

Final recommendations: If used for study, it is recommended that you not read 3 assignments in the time it would take you to read one, but rather, read the same assignment 3 times for exposure and recall improvement, depending on relevancy to testing.

Happy trails, page blazers.

Related and Popular:
How to Tim Ferriss Your Love Life
Tim Ferriss at Princeton (video)



speed per words reading read ]]> Fri, 31 Jul 2009 06:01:08 -0400 30:00 tag:filome.com,10 Cherry-Picker-in-Chief http://www.fivethirtyeight.com/2009/03/obama-ncaa-bracket-shows-bias-toward.html Shared item FiveThirtyEight: Politics Done Right
First shared by - RickKlau
syndication+ 375 | Search 1 | Shares 1

FiveThirtyEight Exclusive: Obama NCAA Bracket Suggests Bias Toward Schools from Swing States



Is President Obama playing favorites with his NCAA bracket? Neither Columbia University nor the University of Chicago have an entry in the Big Dance (the Maroons, alas, aren't even in D-1). Nor does the University of Hawaii or Oregon State University, where Michelle Obama's brother, Craig Robinson, is the coach. The University of Illinois did make the bracket as a #5 seed; Obama has them winning their first round game against Western Kentucky, but then succumbing to the #4-seeded Gonzaga.

There is more than one way, however, for the President to play favorites. Was Obama more inclined to select teams from swing states that were closer in last November's election? A bit of reverse engineering of his bracket suggests that the answer is a qualified yes.

There are 63 games in the men's tournament. In one case, two teams from the same state (Louisville and Morehead State, both from Kentucky) are set to square off. In the other 62 games, Obama has the team from the state which was closer in the November election advancing 35 times (56% of the total).

This in and of itself is not that definitive. Such an outcome could easily have emerged through chance alone. Nor have we accounted for the respective seeds of the teams (the President was highly disinclined to pick underdogs). So a somewhat more rigorous analysis is required.

To do so, I set up a simple logistic regression model containing two independent and one dependent variable. The dependent variable is an indication of whether Obama picked the favorite or the underdog; the variable took on the value of 1 if Obama selected the favorite (that is, the team with the lower seed) and 0 if he took the underdog (the team with the higher seed). In cases where both teams had the same seed (this only occurs once the teams reach the Final 4), the favorite is designated as the team with the higher Associated Press ranking.

The first dependent variable, seeddiff, is the seeding difference between the two clubs. For example, in a game featuring a 7-seed and a 10-seed, seeddiff is 3.

The other dependent variable, swingdiff, is an indication of the relative closeness of the two states in the November election. The swingdiff variable is calculated as follows...

swingdiff = SQRT(margin_underdog) - SQRT(margin_favorite)

...where margin_favorite is the number of points separating Barack Obama and John McCain in the state represented by the favored team, and margin_underdog is the same number for the state represented by the underdog. These values are taken to their square roots because there are diminshing returns once a state becomes non-competitive beyond a certain point; a state which one candidate wins by 20 points isn't really much swingier than one that he won by 30.

For example, take the 1st round game between #6-seeded UCLA and #11-seeded Virginia Commonwealth University. The margin in Virginia on November 4 was 6.3 points, whereas the margin in California was 24.0 points. swingdiff would thus be calculated as follows:

swingdiff = SQRT(6.3) - SQRT(24.0)
swingdiff = 2.51 - 4.90
swingdiff = -2.39

In this case, the value of swingdiff is negative; this indicates that the underdog came from the swingier state. And indeed, Obama picked VCU over UCLA.

When we plug swingdiff and seeddiff into the regression model, we get the following result:



The value for seeddiff is significant and positive, indicating (unsurprisingly) that Obama was more inclined to pick the team with the superior seed, and that this inclination increased with the seeding difference. However, the value for swingdiff is also positive, and significant at the 90 percent probability threshold (although not quite at the 95 percent threshold). It appears probable, therefore, that Obama did in fact have some tendency to prefer teams from swing states in filling out his bracket.

Suppose, for example, that Obama is selecting between a #1-seeded team that comes from a state that he won by 5 points, and a #2-seeded team that comes from a state he won by 20 points. The model predicts that Obama would pick the #1 seed, which happens to be the team from a swing state, about 85 percent of the time.

However, suppose that seedings are reversed: the #2 seed comes from the swing state, whereas the #1 seed does not. In this case, the model predicts that Obama would select the favored #1 seed just 48 pecent of the time, and the swingier #2 seed the rest.

This is actually fairly easy to see if we look at the cases where Obama did, in fact, pick an upset.
1st Round:
Maryland (10) over California (7). Both states are deeply blue, so nothing to work with here.
Tennessee (9) over Oklahoma (8). Likewise, nothing much revealed by this choice between two red state teams, although Tennessee is moderately more swingy than Oklahoma.
Virginia Commonwealth (11) over UCLA (6). Virginia is a swing state, California isn't.
Butler (9) over Louisiana State (8). Butler, based in Indianapolis, is in a swing state whereas LSU isn't.
Temple (11) over Arizona State (6). Although Arizona could plausibly be a swing state with John McCain off the ballot, it's unlikely to have the same import as Pennsylvania, where Temple is located.

2nd Round:
Purdue (5) over Washington (4). Purdue University is located in West Lafayette, Indiana, a swing region in a swing state; Washington isn't really a swing state any longer.
Florida State (5) over Xavier (4). Hard to assign points here; Florida and Ohio (Xavier is located in Cincinnati) are each located in preeminent swing states.

3rd Round:
Syracuse (3) over Oklahoma (2). Deep red state against deep blue state. No net effect, although there are undoubtedly a large number of Syracuse fans in NY-20, which is holding a special election to replace Kirsten Gillibrand later this month.

4th Round:
Memphis (2) over Connecticut (1). Neither Tennessee nor Connecticut is particularly swingy. It's also not clear that this is much of an upset (Memphis is probably the better team).

5th Round:
No upsets.

6th Round (Championship):
North Carolina (1) over Louisville (1). This qualifies as a very mild upset according to AP rankings. North Carolina (where Obama is also friendly with Dean Smith) is a swing state, Kentucky is not.
Although Obama did not pick very many upsets, just about every time he did it tended to favor the team from the swingier state. With that said, there are a couple of upsets that Obama could have reasonably picked but didn't, such as #3-seeded Missouri over #2-seeded Memphis or #2-seeded Michigan State over top seed Louisville.

But yes: the President does appear to have been mixing his hoops with his politics. Will the nation ever recover?


state obama swing seed team ]]>FiveThirtyEight: Politics Done Right
First shared by - RickKlau
syndication+ 375 | Search 1 | Shares 1

FiveThirtyEight Exclusive: Obama NCAA Bracket Suggests Bias Toward Schools from Swing States



Is President Obama playing favorites with his NCAA bracket? Neither Columbia University nor the University of Chicago have an entry in the Big Dance (the Maroons, alas, aren't even in D-1). Nor does the University of Hawaii or Oregon State University, where Michelle Obama's brother, Craig Robinson, is the coach. The University of Illinois did make the bracket as a #5 seed; Obama has them winning their first round game against Western Kentucky, but then succumbing to the #4-seeded Gonzaga.

There is more than one way, however, for the President to play favorites. Was Obama more inclined to select teams from swing states that were closer in last November's election? A bit of reverse engineering of his bracket suggests that the answer is a qualified yes.

There are 63 games in the men's tournament. In one case, two teams from the same state (Louisville and Morehead State, both from Kentucky) are set to square off. In the other 62 games, Obama has the team from the state which was closer in the November election advancing 35 times (56% of the total).

This in and of itself is not that definitive. Such an outcome could easily have emerged through chance alone. Nor have we accounted for the respective seeds of the teams (the President was highly disinclined to pick underdogs). So a somewhat more rigorous analysis is required.

To do so, I set up a simple logistic regression model containing two independent and one dependent variable. The dependent variable is an indication of whether Obama picked the favorite or the underdog; the variable took on the value of 1 if Obama selected the favorite (that is, the team with the lower seed) and 0 if he took the underdog (the team with the higher seed). In cases where both teams had the same seed (this only occurs once the teams reach the Final 4), the favorite is designated as the team with the higher Associated Press ranking.

The first dependent variable, seeddiff, is the seeding difference between the two clubs. For example, in a game featuring a 7-seed and a 10-seed, seeddiff is 3.

The other dependent variable, swingdiff, is an indication of the relative closeness of the two states in the November election. The swingdiff variable is calculated as follows...

swingdiff = SQRT(margin_underdog) - SQRT(margin_favorite)

...where margin_favorite is the number of points separating Barack Obama and John McCain in the state represented by the favored team, and margin_underdog is the same number for the state represented by the underdog. These values are taken to their square roots because there are diminshing returns once a state becomes non-competitive beyond a certain point; a state which one candidate wins by 20 points isn't really much swingier than one that he won by 30.

For example, take the 1st round game between #6-seeded UCLA and #11-seeded Virginia Commonwealth University. The margin in Virginia on November 4 was 6.3 points, whereas the margin in California was 24.0 points. swingdiff would thus be calculated as follows:

swingdiff = SQRT(6.3) - SQRT(24.0)
swingdiff = 2.51 - 4.90
swingdiff = -2.39

In this case, the value of swingdiff is negative; this indicates that the underdog came from the swingier state. And indeed, Obama picked VCU over UCLA.

When we plug swingdiff and seeddiff into the regression model, we get the following result:



The value for seeddiff is significant and positive, indicating (unsurprisingly) that Obama was more inclined to pick the team with the superior seed, and that this inclination increased with the seeding difference. However, the value for swingdiff is also positive, and significant at the 90 percent probability threshold (although not quite at the 95 percent threshold). It appears probable, therefore, that Obama did in fact have some tendency to prefer teams from swing states in filling out his bracket.

Suppose, for example, that Obama is selecting between a #1-seeded team that comes from a state that he won by 5 points, and a #2-seeded team that comes from a state he won by 20 points. The model predicts that Obama would pick the #1 seed, which happens to be the team from a swing state, about 85 percent of the time.

However, suppose that seedings are reversed: the #2 seed comes from the swing state, whereas the #1 seed does not. In this case, the model predicts that Obama would select the favored #1 seed just 48 pecent of the time, and the swingier #2 seed the rest.

This is actually fairly easy to see if we look at the cases where Obama did, in fact, pick an upset.
1st Round:
Maryland (10) over California (7). Both states are deeply blue, so nothing to work with here.
Tennessee (9) over Oklahoma (8). Likewise, nothing much revealed by this choice between two red state teams, although Tennessee is moderately more swingy than Oklahoma.
Virginia Commonwealth (11) over UCLA (6). Virginia is a swing state, California isn't.
Butler (9) over Louisiana State (8). Butler, based in Indianapolis, is in a swing state whereas LSU isn't.
Temple (11) over Arizona State (6). Although Arizona could plausibly be a swing state with John McCain off the ballot, it's unlikely to have the same import as Pennsylvania, where Temple is located.

2nd Round:
Purdue (5) over Washington (4). Purdue University is located in West Lafayette, Indiana, a swing region in a swing state; Washington isn't really a swing state any longer.
Florida State (5) over Xavier (4). Hard to assign points here; Florida and Ohio (Xavier is located in Cincinnati) are each located in preeminent swing states.

3rd Round:
Syracuse (3) over Oklahoma (2). Deep red state against deep blue state. No net effect, although there are undoubtedly a large number of Syracuse fans in NY-20, which is holding a special election to replace Kirsten Gillibrand later this month.

4th Round:
Memphis (2) over Connecticut (1). Neither Tennessee nor Connecticut is particularly swingy. It's also not clear that this is much of an upset (Memphis is probably the better team).

5th Round:
No upsets.

6th Round (Championship):
North Carolina (1) over Louisville (1). This qualifies as a very mild upset according to AP rankings. North Carolina (where Obama is also friendly with Dean Smith) is a swing state, Kentucky is not.
Although Obama did not pick very many upsets, just about every time he did it tended to favor the team from the swingier state. With that said, there are a couple of upsets that Obama could have reasonably picked but didn't, such as #3-seeded Missouri over #2-seeded Memphis or #2-seeded Michigan State over top seed Louisville.

But yes: the President does appear to have been mixing his hoops with his politics. Will the nation ever recover?


state obama swing seed team ]]> Thu, 19 Mar 2009 12:42:36 -0400 30:00 tag:filome.com,11 Suspension of Dividends http://feedproxy.google.com/~r/LongOrShort/~3/N5cU-D6waR4/suspension-of-dividends.htm Shared item Long or Short Capital
First shared by - RickKlau
syndication+ 82 | Search 1 | Shares 1

Like most of you, Long or Short Capital is not entirely immune from the current dislocation going on in capital markets. Also like most of you, Long or Short Capital recognizes that this period of extended dislocation and, more generally, the backdrop of the Great Regression, provide the perfect cover for past acts of corporate malfeasance and accounting shenanigans.

The good news is that you, as subcriberholders under our former dividend policy, have been compensated both in the form of money, and in ithe form insightful investable recommendations. But there is bad news. Due to the current capital market uncertainty, especially the difficulty we have encountered in trying to secure a revolving line of credit so that we are able maintain sufficient liquidity, as well as the difficulty we have had in finding (entirely theoretical) potential debtor-in-possession financing (seriously we aren't going to file, we swear on Madison's life), we have entered into a cash conservation mode. We will be suspending our dividends, retroactively and futurely.

We have hired a financial advisor to explore our options both with respect both to our capital structure and to maximizing value to our executive team. We have them looking into the appropriateness of soliciting reverse dividends from our subscriberholders so as to improve our liquidity and ease the burden of certain off-balance sheet arrangements that were undertaken to finance my yacht (sometimes referred to as the corporate yacht in our filings), as well as my summer yacht.

We appreciate your continued support in these trying times. Personally, I think you can relate to the humiliating devastation I feel as I am forced to go from one mega-yacht I purchased due to my excessive compensation as CEO, one company yacht (in name only, I used it exclusively), and one summer yacht paid for by embezzling LoS capital funds, to only one mega-yacht.

For updates on our dividend policy, always navigate to our Dividend Policy. Thank you.



yacht capital dividends policy dividend ]]>Long or Short Capital
First shared by - RickKlau
syndication+ 82 | Search 1 | Shares 1

Like most of you, Long or Short Capital is not entirely immune from the current dislocation going on in capital markets. Also like most of you, Long or Short Capital recognizes that this period of extended dislocation and, more generally, the backdrop of the Great Regression, provide the perfect cover for past acts of corporate malfeasance and accounting shenanigans.

The good news is that you, as subcriberholders under our former dividend policy, have been compensated both in the form of money, and in ithe form insightful investable recommendations. But there is bad news. Due to the current capital market uncertainty, especially the difficulty we have encountered in trying to secure a revolving line of credit so that we are able maintain sufficient liquidity, as well as the difficulty we have had in finding (entirely theoretical) potential debtor-in-possession financing (seriously we aren't going to file, we swear on Madison's life), we have entered into a cash conservation mode. We will be suspending our dividends, retroactively and futurely.

We have hired a financial advisor to explore our options both with respect both to our capital structure and to maximizing value to our executive team. We have them looking into the appropriateness of soliciting reverse dividends from our subscriberholders so as to improve our liquidity and ease the burden of certain off-balance sheet arrangements that were undertaken to finance my yacht (sometimes referred to as the corporate yacht in our filings), as well as my summer yacht.

We appreciate your continued support in these trying times. Personally, I think you can relate to the humiliating devastation I feel as I am forced to go from one mega-yacht I purchased due to my excessive compensation as CEO, one company yacht (in name only, I used it exclusively), and one summer yacht paid for by embezzling LoS capital funds, to only one mega-yacht.

For updates on our dividend policy, always navigate to our Dividend Policy. Thank you.



yacht capital dividends policy dividend ]]> Tue, 10 Feb 2009 12:18:37 -0500 30:00 tag:filome.com,12 Projection: Franken to Win Recount by 27 Votes http://www.fivethirtyeight.com/2008/11/projection-franken-to-win-recount-by-27.html Shared item FiveThirtyEight.com: Politics Done Right
First shared by - RickKlau
syndication+ 127 | Search 1 | Shares 1

As we wrote yesterday evening, the ever-increasing number of challenged ballots in Minnesota is making it more and more difficult to determine the extent to which Al Franken is in fact gaining ground in the state's recount process. An analysis of precinct-by-precinct returns available on the Secretary of State's website, however, suggests that Franken's position is somewhat stronger than it appears, and that he may in fact be the favorite to prevail in the recount process.

Consider the following. In precincts where no challenges have been issued (these are the only precincts in which, in some sense, the results of the recount can be considered to be final and "official") Franken has gained a total of 34 votes, and Coleman a total of 6 votes, for a net gain by Franken of 28 votes. Moreover, in precincts where just 1 challenge has been issued, Franken has gained a net of 31 votes on Coleman, and in precincts where exactly 2 challenges have been issued, Franken has gained a net of 32 votes on Coleman.

By contrast, in precincts where 5 or more ballots have been challenged between the two campaigns, Coleman has gained a net of 57 votes on Franken.

In other words, the fewer the number of challenged ballots, the better Franken is doing, and the higher the number of challenged ballots, the worse he is doing; the relationship is in fact quite strong.
Precinct-Level Returns Analysis
# Challenges    n      Franken    Coleman     Net
0               2233   +34        +6          Franken +28
1               419    -94        -125        Franken +31
2               154    -90        -122        Franken +32
3-4             133    -157       -171        Franken +14
5-9             59     -158       -116        Coleman -42
10+             26     -156       -141        Coleman -15
It is not an accident, then, that as the number of challenges has increased with each day of the recount, Franken's momentum appears to have stalled out. Very probably, a majority of the challenges are coming from Franken's pile. This is somewhat irrespective of which campaign actually instigates the challenge, since as we suggested yesterday, a potential Franken undervote could be the subject of a challenge from either campaign depending on the initial ruling of the local elections judge.

We can address this phenomenon more systematically by means of a regression analysis. In the regression, we are attempting to predict a variable I've defined as franken_net, which is the net gain by Franken per 10,000 ballots cast in that precinct. The independent variables considered in the regression are as follows:

t: the proportion of the two-way vote received by Franken in the initial count (e.g. excluding votes for third parties)

c_f: the number of challenges initiated by the Franken campaign per 10,000 ballots counted in that precinct

c_c: the number of challenges initiated by the Coleman campaign per 10,000 ballots counted in that precinct

In addition, the regression analysis contains interaction terms between each combination of two variables, as well as an interaction term for all three variables, all of which are statistically significant. The regression is weighted by the square root of the number of ballots cast in that precinct.

The results of the regression are as follows:
franken_net        Coef.     t     P>|t|
t                  8.922     2.89  0.004
c_f               -0.280    -3.99  0.000
c_c               -0.926    -9.82  0.000
t * c_f           -0.703    -8.59  0.000
t * c_c           +0.565     2.89  0.004
c_f * c_c         -0.013    -4.29  0.000
t * c_f * c_c     +0.012     2.81  0.005
_constant         -3.622    -2.36  0.019
This regression is a bit difficult to interpret, particularly with the presence of all the interaction terms, but the key intuition is as follows. Suppose that the number of challenges is zero -- as will happen once the state canvassing board finishes considering all such challenges in December. In this case, all terms in the regression equation reduce to zero, except for the constant term and t, which is Franken's share of the two-way vote in that precinct. We are thus left with the following:

franken_net = t * 8.922 - 3.622

Now, we can attempt to solve this equation at the statewide level. When we plug in a t of .499956 -- Franken was picked on just slightly very less than half of the ballots during the initial count -- we get a value for franken_net of .837. That is, Franken will gain a net of .837 votes for every 10,000 cast. With a total of 2,885,555 ballots having been recorded in the initial count, this works out to a projected gain of 242 votes for Franken statewide. Since Norm Coleman led by 215 votes in the initial count, this suggests that Franken will win by 27 votes once the recount process is complete (including specifically the adjudication of all challenged ballots).

The error bars on this regression analysis are fairly high, and so even if you buy my analysis, you should not regard Franken as more than a very slight favorite. Nevertheless, there is good reason to believe that the high rate of ballot challenges is in fact hurting Franken disproportionately, and that once such challenges are resolved, Franken stands to gain ground, perhaps enough to let him overtake Coleman.

(Note: it is also possible to build a multivariate regression model that attempts to solve for both Franken and Coleman's totals in an absolute sense, rather than Franken's gain relative to Coleman. This multivariate model produces a slightly more optimistic result for Franken, suggesting that he will gain 254 votes statewide and Coleman will lose 12, producing a net swing of 268 votes toward Franken.)

franken c votes coleman net ]]>FiveThirtyEight.com: Politics Done Right
First shared by - RickKlau
syndication+ 127 | Search 1 | Shares 1

As we wrote yesterday evening, the ever-increasing number of challenged ballots in Minnesota is making it more and more difficult to determine the extent to which Al Franken is in fact gaining ground in the state's recount process. An analysis of precinct-by-precinct returns available on the Secretary of State's website, however, suggests that Franken's position is somewhat stronger than it appears, and that he may in fact be the favorite to prevail in the recount process.

Consider the following. In precincts where no challenges have been issued (these are the only precincts in which, in some sense, the results of the recount can be considered to be final and "official") Franken has gained a total of 34 votes, and Coleman a total of 6 votes, for a net gain by Franken of 28 votes. Moreover, in precincts where just 1 challenge has been issued, Franken has gained a net of 31 votes on Coleman, and in precincts where exactly 2 challenges have been issued, Franken has gained a net of 32 votes on Coleman.

By contrast, in precincts where 5 or more ballots have been challenged between the two campaigns, Coleman has gained a net of 57 votes on Franken.

In other words, the fewer the number of challenged ballots, the better Franken is doing, and the higher the number of challenged ballots, the worse he is doing; the relationship is in fact quite strong.
Precinct-Level Returns Analysis
# Challenges    n      Franken    Coleman     Net
0               2233   +34        +6          Franken +28
1               419    -94        -125        Franken +31
2               154    -90        -122        Franken +32
3-4             133    -157       -171        Franken +14
5-9             59     -158       -116        Coleman -42
10+             26     -156       -141        Coleman -15
It is not an accident, then, that as the number of challenges has increased with each day of the recount, Franken's momentum appears to have stalled out. Very probably, a majority of the challenges are coming from Franken's pile. This is somewhat irrespective of which campaign actually instigates the challenge, since as we suggested yesterday, a potential Franken undervote could be the subject of a challenge from either campaign depending on the initial ruling of the local elections judge.

We can address this phenomenon more systematically by means of a regression analysis. In the regression, we are attempting to predict a variable I've defined as franken_net, which is the net gain by Franken per 10,000 ballots cast in that precinct. The independent variables considered in the regression are as follows:

t: the proportion of the two-way vote received by Franken in the initial count (e.g. excluding votes for third parties)

c_f: the number of challenges initiated by the Franken campaign per 10,000 ballots counted in that precinct

c_c: the number of challenges initiated by the Coleman campaign per 10,000 ballots counted in that precinct

In addition, the regression analysis contains interaction terms between each combination of two variables, as well as an interaction term for all three variables, all of which are statistically significant. The regression is weighted by the square root of the number of ballots cast in that precinct.

The results of the regression are as follows:
franken_net        Coef.     t     P>|t|
t                  8.922     2.89  0.004
c_f               -0.280    -3.99  0.000
c_c               -0.926    -9.82  0.000
t * c_f           -0.703    -8.59  0.000
t * c_c           +0.565     2.89  0.004
c_f * c_c         -0.013    -4.29  0.000
t * c_f * c_c     +0.012     2.81  0.005
_constant         -3.622    -2.36  0.019
This regression is a bit difficult to interpret, particularly with the presence of all the interaction terms, but the key intuition is as follows. Suppose that the number of challenges is zero -- as will happen once the state canvassing board finishes considering all such challenges in December. In this case, all terms in the regression equation reduce to zero, except for the constant term and t, which is Franken's share of the two-way vote in that precinct. We are thus left with the following:

franken_net = t * 8.922 - 3.622

Now, we can attempt to solve this equation at the statewide level. When we plug in a t of .499956 -- Franken was picked on just slightly very less than half of the ballots during the initial count -- we get a value for franken_net of .837. That is, Franken will gain a net of .837 votes for every 10,000 cast. With a total of 2,885,555 ballots having been recorded in the initial count, this works out to a projected gain of 242 votes for Franken statewide. Since Norm Coleman led by 215 votes in the initial count, this suggests that Franken will win by 27 votes once the recount process is complete (including specifically the adjudication of all challenged ballots).

The error bars on this regression analysis are fairly high, and so even if you buy my analysis, you should not regard Franken as more than a very slight favorite. Nevertheless, there is good reason to believe that the high rate of ballot challenges is in fact hurting Franken disproportionately, and that once such challenges are resolved, Franken stands to gain ground, perhaps enough to let him overtake Coleman.

(Note: it is also possible to build a multivariate regression model that attempts to solve for both Franken and Coleman's totals in an absolute sense, rather than Franken's gain relative to Coleman. This multivariate model produces a slightly more optimistic result for Franken, suggesting that he will gain 254 votes statewide and Coleman will lose 12, producing a net swing of 268 votes toward Franken.)

franken c votes coleman net ]]> Sun, 23 Nov 2008 15:06:28 -0500 30:00 tag:filome.com,13 Allocating the Undecideds http://www.fivethirtyeight.com/2008/09/allocating-undecideds.html Shared item FiveThirtyEight.com: Electoral Projections Done Right
First shared by - RickKlau
syndication+ 148 | Search 65 | Shares 1

Heretofore, I've simply been allocating undecided voters 50:50. That is certainly the most neutral assumption to make. But this site isn't about making the most neutral assumption, it's about making the most predictive one.

So what I'm going to do instead is tie the undecided allocation to the extent to which Barack Obama overperformed or underperformed his polls in particular types of states in the Democratic primaries. If you compare the actual results in the primaries to the final RCP or Pollster.com averages, you'll notice some fairly systematic differences.

Specifically, Obama overperformed:

1. In states with high African-American populations;
2. In states that share a border with Illinois (no, Kentucky doesn't count);
3. In states with highly educated electorates;
4. To a lesser extent, in the South (as indicated by the number of evangelicals), even after accounting for the above variables.

Meanwhile, he underperformed his polls:

1. In the Appalachian states (as indicated by the number of respondents who identify their ancestry as 'American', a practice concentrated in the Appalachian region);
2. In states with low education levels;
3. And in states with a high number of Catholics.

This can all be ferreted out via regression analysis, taking the factors I describe above as the independent variables, and Obama's performance vis--vis his polls as the dependent variable. The R-squared on this regression is .72, which is quite high -- it means that it was rather predictable when the polls were wrong, and in which direction.

To get a little ahead of myself: does this mean that there was in fact a Bradley Effect during the primaries? It's not clear. What is actually quite clear -- and I'm going to present some research on this over the next several days -- is that the polls did a rather poor job of accounting for the black vote. Not only did essentially every "undecided" African-American voter wind up voting for Obama, but some of those who told pollsters they were going to vote for Hillary also wound up voting for Obama. The reverse Bradley Effect, in other words, was fairly manifest.

It's also clear that there were some patterns in the way that undecided white voters behaved. Number one, a majority of them -- probably somewhere between 60 and 65 percent -- wound up voting for Clinton. This is perhaps not so remarkable, considering that about 60 percent of white voters in the primaries voted for Clinton period. But, this figure was higher in regions like the Appalachians, and among groups like Catholics, and lower in places where you had a lot of WASPy, educated voters. So whether or not you label this a Bradley Effect, I don't know -- but the behavior of undecided voters has been predictable to a certain extent.

Now, it does not necessarily follow that the patterns exhibited by undecided voters in the primaries will match those in the general election. But based both on my research and on what I've been hearing from people on the ground, it's apparent that the public polling in general is not terrific, and that if we have an instinct about where the polls are more likely to come in high or low, we probably ought to follow it.

So what I've done is to transform the results of the regression analysis that I described above into an undecided voter allocation for each state. The allocation is "rigged" such that neither candidate will gain or lose ground in the national popular vote as a result, and such that the range of allocations runs from about .35 to .65. That is, in some states we'll allocate as much as 65% of the undecided vote to John McCain (and just 35% to Barack Obama) and in others we'll allocate as much as 65% to Obama (and just 35% to McCain).

The specific allocations follow. Remember, these are based on the extent to which Obama over- or underperformed his polls in various states during the primaries:

Percent of Undecided Votes Allocated to Barack Obama
DC 64.4%
MS 64.4%
GA 63.0%
MD 61.5%
SC 61.1%
AL 60.9%
NC 58.0%
VA 57.8%
IN 57.8%
IA 56.9%
AR 56.8%
OK 56.5%
WI 56.5%
DE 53.7%
AK 53.4%
WA 52.7%
FL 52.4%
TN 52.3%
CO 51.8%
MO 51.6%
MI 51.5%
KS 51.4%
OR 51.0%
LA 50.7%
UT 50.6%
HI 50.5%
MN 50.2%
NE 49.8%
TX 48.3%
IL 48.3%
MT 48.0%
OH 47.2%
NV 46.7%
WY 46.6%
SD 46.4%
AZ 46.0%
ND 45.5%
ID 45.4%
NJ 45.0%
PA 44.8%
CT 44.6%
NY 44.6%
VT 43.7%
KY 43.4%
CA 42.9%
ME 42.6%
NH 42.2%
MA 41.0%
NM 40.0%
WV 38.6%
RI 35.0%

At this point in the election, the number of undecideds is fairly low: generally between 4 and 6 points in each state, once we've gotten done assigning a point or two to third party candidates. As such, these allocations do not make a great deal of difference -- at the most, a swing of maybe a point or a point-and-a-half.

Still, you can see some impacts at the margins. Take a state like West Virginia, where the polling has been reasonably close but where there are also high numbers of undecided voters. Those undecideds aren't the type of undecideds who are liable to side with Barack Obama when pushed to a decision, and so the state is not quite as promising for him as it looks on paper. There are also a fairly high number of undecideds in Ohio, a state where we think the undecided vote is liable to break slightly for John McCain. On the other hand, a state like Virginia, where Obama overperformed his polls during the primaries and where some polling has had a relatively generous (and probably false) number of African-American votes going to John McCain, might be just a smidgen stronger for Obama than it appears.

obama undecided states polls primaries ]]>FiveThirtyEight.com: Electoral Projections Done Right
First shared by - RickKlau
syndication+ 148 | Search 65 | Shares 1

Heretofore, I've simply been allocating undecided voters 50:50. That is certainly the most neutral assumption to make. But this site isn't about making the most neutral assumption, it's about making the most predictive one.

So what I'm going to do instead is tie the undecided allocation to the extent to which Barack Obama overperformed or underperformed his polls in particular types of states in the Democratic primaries. If you compare the actual results in the primaries to the final RCP or Pollster.com averages, you'll notice some fairly systematic differences.

Specifically, Obama overperformed:

1. In states with high African-American populations;
2. In states that share a border with Illinois (no, Kentucky doesn't count);
3. In states with highly educated electorates;
4. To a lesser extent, in the South (as indicated by the number of evangelicals), even after accounting for the above variables.

Meanwhile, he underperformed his polls:

1. In the Appalachian states (as indicated by the number of respondents who identify their ancestry as 'American', a practice concentrated in the Appalachian region);
2. In states with low education levels;
3. And in states with a high number of Catholics.

This can all be ferreted out via regression analysis, taking the factors I describe above as the independent variables, and Obama's performance vis--vis his polls as the dependent variable. The R-squared on this regression is .72, which is quite high -- it means that it was rather predictable when the polls were wrong, and in which direction.

To get a little ahead of myself: does this mean that there was in fact a Bradley Effect during the primaries? It's not clear. What is actually quite clear -- and I'm going to present some research on this over the next several days -- is that the polls did a rather poor job of accounting for the black vote. Not only did essentially every "undecided" African-American voter wind up voting for Obama, but some of those who told pollsters they were going to vote for Hillary also wound up voting for Obama. The reverse Bradley Effect, in other words, was fairly manifest.

It's also clear that there were some patterns in the way that undecided white voters behaved. Number one, a majority of them -- probably somewhere between 60 and 65 percent -- wound up voting for Clinton. This is perhaps not so remarkable, considering that about 60 percent of white voters in the primaries voted for Clinton period. But, this figure was higher in regions like the Appalachians, and among groups like Catholics, and lower in places where you had a lot of WASPy, educated voters. So whether or not you label this a Bradley Effect, I don't know -- but the behavior of undecided voters has been predictable to a certain extent.

Now, it does not necessarily follow that the patterns exhibited by undecided voters in the primaries will match those in the general election. But based both on my research and on what I've been hearing from people on the ground, it's apparent that the public polling in general is not terrific, and that if we have an instinct about where the polls are more likely to come in high or low, we probably ought to follow it.

So what I've done is to transform the results of the regression analysis that I described above into an undecided voter allocation for each state. The allocation is "rigged" such that neither candidate will gain or lose ground in the national popular vote as a result, and such that the range of allocations runs from about .35 to .65. That is, in some states we'll allocate as much as 65% of the undecided vote to John McCain (and just 35% to Barack Obama) and in others we'll allocate as much as 65% to Obama (and just 35% to McCain).

The specific allocations follow. Remember, these are based on the extent to which Obama over- or underperformed his polls in various states during the primaries:

Percent of Undecided Votes Allocated to Barack Obama
DC 64.4%
MS 64.4%
GA 63.0%
MD 61.5%
SC 61.1%
AL 60.9%
NC 58.0%
VA 57.8%
IN 57.8%
IA 56.9%
AR 56.8%
OK 56.5%
WI 56.5%
DE 53.7%
AK 53.4%
WA 52.7%
FL 52.4%
TN 52.3%
CO 51.8%
MO 51.6%
MI 51.5%
KS 51.4%
OR 51.0%
LA 50.7%
UT 50.6%
HI 50.5%
MN 50.2%
NE 49.8%
TX 48.3%
IL 48.3%
MT 48.0%
OH 47.2%
NV 46.7%
WY 46.6%
SD 46.4%
AZ 46.0%
ND 45.5%
ID 45.4%
NJ 45.0%
PA 44.8%
CT 44.6%
NY 44.6%
VT 43.7%
KY 43.4%
CA 42.9%
ME 42.6%
NH 42.2%
MA 41.0%
NM 40.0%
WV 38.6%
RI 35.0%

At this point in the election, the number of undecideds is fairly low: generally between 4 and 6 points in each state, once we've gotten done assigning a point or two to third party candidates. As such, these allocations do not make a great deal of difference -- at the most, a swing of maybe a point or a point-and-a-half.

Still, you can see some impacts at the margins. Take a state like West Virginia, where the polling has been reasonably close but where there are also high numbers of undecided voters. Those undecideds aren't the type of undecideds who are liable to side with Barack Obama when pushed to a decision, and so the state is not quite as promising for him as it looks on paper. There are also a fairly high number of undecideds in Ohio, a state where we think the undecided vote is liable to break slightly for John McCain. On the other hand, a state like Virginia, where Obama overperformed his polls during the primaries and where some polling has had a relatively generous (and probably false) number of African-American votes going to John McCain, might be just a smidgen stronger for Obama than it appears.

obama undecided states polls primaries ]]> Mon, 22 Sep 2008 10:38:19 -0400 30:00 tag:filome.com,14