![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
athelindThere's been a bit of a kerfluffle about a recent study about students who fell for a hoax website about the Pacific Northwest Tree Octopus.
Frankly, the article linked above is a shoddy piece of science journalism. As![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
eggshellhammer pointed out, it doesn't link to the original study. Even worse, in Your Obedient Serpent's eyes: it didn't specify the age level of the students. That's an important factor: a study about the critical thinking ability of kindergarten students has entirely different implications than the same study about a group of college undergraduates.
That in itself is an indication of a failure of critical thinking ability in would-be science journalists.
As it transpires, this study involved seventh-graders. The conclusion can thus be summarized as, "wow, you can con a 12-year-old into believing some crazy shit", which is hardly earth-shattering news. I'd say three-quarters of the contents of snopes.com is stuff that was repeated as gospel truth on the Bicentennial schoolyards of my twelfth year.
(I find the datum that students ignore search engines in favor of randomlytypinginaname.com to be much more startling, personally. Seriously, WTF?)
The other study mentioned in the University of Connecticut article suggests that this, in large measure, just reflects a need for improved emphasis on Internet search and access skills, and not some Terrible Crisis in Education. That's how the researchers seem to interpret it; the DANGER WILL ROBINSON! reactions were mostly imposed by the secondary sources. For my part, I was intrigued and, on some level, amused at the revelation that students who had difficulties with traditional literacy showed superior online reading facilities.
As for the details of the first study ... I'm going to be generous and completely ignore the implications of drawing broad conclusions from a sample group of twenty-five students in a single class. Let's assume, for the sake of argument, that this specific class is representative of the entire population of students in Connecticut. Let's take a look at two of the sited conclusions:
• All but one of the 25 rated the site as "very credible" ...
Let us, just for a moment, step out of the role of of the Know-It-All Grown-Up Who Knows This Site Is Patently Absurd Because There's No Such Thing. Let us remember that those reading this journal are likely to have at least five more years of formal education than the subjects of this study.
Yes, http://zapatopi.net/treeoctopus/ is "very credible".
"Credible" doesn't mean "true" or "accurate". It means "able to be believed", or "capable of persuading". The website has a professional presentation and a serious, convincing tone. The only obvious joke on the main page (aside from a deadpan link to sasquatch) is a reference to the organization "Greenpeas". The FAQ gets increasingly flippant and absurdist, but they avoid an overtly humorous tone for the main body.
Given that aquarium octopuses are well-known for getting out of their tanks and taking walks, and that there is at least one species of land-dwelling, arboreal hermit crabs, the idea of a "tree octopus" is just plausible enough to someone who knows just how weird and wacky life on Earth can get.
In science, "credibility" also means "reproducibility", and in this context, that extends to being able to find other corroborating sources.
This leads us to the second conclusion I want to examine:
• Most struggled when asked to produce proof - or even clues - that the web site was false ...
Hey, it's an exercise for the class! Let's check our own research and critical thinking abilities, shall we?
I'm curious to see what proofs (or even clues!) the folks reading this can come up with, above and beyond the flippant tone of the FAQ that I mentioned above. The Sasquatch link leads to an equally-deadpan page, of course.
Needless to say, "I just know there's no such thing" isn't a valid "proof"; in fact, it doesn't even rate as a "clue".
Answers will be graded!
Thanks ... and apologies ... topseudomanitou for drawing my attention to this study and the reactions which followed. Please don't think I'm being an asshole for deconstructing this.
Update:eggshellhammer contacted the original author and scored a link to the original document. Yes, the sample group was larger than 25.
Frankly, the article linked above is a shoddy piece of science journalism. As
eggshellhammer pointed out, it doesn't link to the original study. Even worse, in Your Obedient Serpent's eyes: it didn't specify the age level of the students. That's an important factor: a study about the critical thinking ability of kindergarten students has entirely different implications than the same study about a group of college undergraduates. That in itself is an indication of a failure of critical thinking ability in would-be science journalists.
As it transpires, this study involved seventh-graders. The conclusion can thus be summarized as, "wow, you can con a 12-year-old into believing some crazy shit", which is hardly earth-shattering news. I'd say three-quarters of the contents of snopes.com is stuff that was repeated as gospel truth on the Bicentennial schoolyards of my twelfth year.
(I find the datum that students ignore search engines in favor of randomlytypinginaname.com to be much more startling, personally. Seriously, WTF?)
The other study mentioned in the University of Connecticut article suggests that this, in large measure, just reflects a need for improved emphasis on Internet search and access skills, and not some Terrible Crisis in Education. That's how the researchers seem to interpret it; the DANGER WILL ROBINSON! reactions were mostly imposed by the secondary sources. For my part, I was intrigued and, on some level, amused at the revelation that students who had difficulties with traditional literacy showed superior online reading facilities.
As for the details of the first study ... I'm going to be generous and completely ignore the implications of drawing broad conclusions from a sample group of twenty-five students in a single class. Let's assume, for the sake of argument, that this specific class is representative of the entire population of students in Connecticut. Let's take a look at two of the sited conclusions:
• All but one of the 25 rated the site as "very credible" ...
Let us, just for a moment, step out of the role of of the Know-It-All Grown-Up Who Knows This Site Is Patently Absurd Because There's No Such Thing. Let us remember that those reading this journal are likely to have at least five more years of formal education than the subjects of this study.
Yes, http://zapatopi.net/treeoctopus/ is "very credible".
"Credible" doesn't mean "true" or "accurate". It means "able to be believed", or "capable of persuading". The website has a professional presentation and a serious, convincing tone. The only obvious joke on the main page (aside from a deadpan link to sasquatch) is a reference to the organization "Greenpeas". The FAQ gets increasingly flippant and absurdist, but they avoid an overtly humorous tone for the main body.
Given that aquarium octopuses are well-known for getting out of their tanks and taking walks, and that there is at least one species of land-dwelling, arboreal hermit crabs, the idea of a "tree octopus" is just plausible enough to someone who knows just how weird and wacky life on Earth can get.
In science, "credibility" also means "reproducibility", and in this context, that extends to being able to find other corroborating sources.
This leads us to the second conclusion I want to examine:
• Most struggled when asked to produce proof - or even clues - that the web site was false ...
Hey, it's an exercise for the class! Let's check our own research and critical thinking abilities, shall we?
I'm curious to see what proofs (or even clues!) the folks reading this can come up with, above and beyond the flippant tone of the FAQ that I mentioned above. The Sasquatch link leads to an equally-deadpan page, of course.
Needless to say, "I just know there's no such thing" isn't a valid "proof"; in fact, it doesn't even rate as a "clue".
Answers will be graded!
Thanks ... and apologies ... to
pseudomanitou for drawing my attention to this study and the reactions which followed. Please don't think I'm being an asshole for deconstructing this.Update:
eggshellhammer contacted the original author and scored a link to the original document. Yes, the sample group was larger than 25.
Re: Proving the website is false would be tricky
Date: 2011-02-04 04:10 pm (UTC)One way of searching for "all the things" is to use available references, especially those which are exhaustive (meaning they say they list All The Things, as Encyclopedia Britannica and Wikipedia do). If there is no tree octopus in those references, which may be more or less exhaustive, then one may safely assume that the tree octopus is either a newly-found species or is not an Earthly creature (one cannot say that a tree octopus does not exist elsewhere, but then, one cannot also yet say that life does or does not exist elsewhere using current scientific knowledge).
Adding to one's search for references by looking for scientific papers and websites outside of the single given site will also help to confirm or deny, as will looking over the other pages on the cited site itself.
Re: Proving the website is false would be tricky
Date: 2011-02-05 05:24 am (UTC)Axiom 1: If Wikipedia lists all the things, then it lists me.
Axiom 1.1: I am a member of the set of all the things.
Axiom 2: If something is listed in Wikipedia, Wikipedia's search function will reveal it.
Axiom 2.1: Wikipedia's search function is without flaws.
Wikipedia's search function does not reveal a listing for me, therefore I am not listed in Wikipedia. Since I am not listed in Wikipedia, either I am not a member of the set of all things (contradicting axiom 2.1), Wikipedia's search function does not cover all of Wikipedia's content (contradicting axiom 2 and 2.1), or Wikipedia does not list all things (contradicting axiom 1).
But yeah, I'm generally content to look it up in a few places, maybe go check in person if it's not the other side of the country, and call it a day.
Re: Proving the website is false would be tricky
Date: 2011-02-05 08:36 am (UTC)One thing that jumped out at me was the passage that listed reasons for the low populations of the Pacific Northwest Tree Octopus—and included among those reasons .
That's an assertion about a species that is well-known to be itself endangered, not a booming population. If I were taken in by the rest of the site, missed the "Greenpeas" reference, and wasn't sure about the veracity of the sasquatch links, that would have gotten my attention. It is also an assertion that is quickly and easily testable with just a quick Google search.
Of course, that same "quick Google search" on "Pacific Tree Octopus" or "Octopus paxarbolis" will immediately yield several sites asserting that the original site is a hoax, but that's so simple it's cheating.