Minichan

Well... Okay. You seem like you're being sincere, so I'll post my thoughts.

Yeah, I have no problem admitting that it is an issue in the field. When evaluating the sources I have gathered for an essay, I find that I can often put in "low population validity" (meaning that it is not representative) as a criticism. Here, I found an interesting article that explains the problem quite clearly and concisely. Basically, it happens because college students are convinient, and researchers and journals tolerate it.

> but do you have any qualms about so many of your field's bedrock principles being based on questionable studies
Am a little unsure at what you're getting at, here. If I'm understanding right, you're saying that some of Psychology's most fundamental theories may be incorrect due to questionable methodology? If so, I'll say that the "bedrock" stuff has likely been retested in various forms all throughout the world and over an exteneded period of time. While I do think that some research may be adveresly affected by poorly representative participant samples, I'm confident that the field has a good handle on the "bedrock" stuff.

Additionally, human behaviour is undoubtedly a topic worthy of study and with lots of potential benefit. Whilst this is an worrying issue within the field, I think that researchers generally do a good job of testing and applying theories. And, needless to say, hopefully this will improve in time.

@previous (Killer Lettuce !!iNo3FkiZx)

> Well... Okay. You seem like you're being sincere, so I'll post my thoughts.

I am!

> > but do you have any qualms about so many of your field's bedrock principles being based on questionable studies
> Am a little unsure at what you're getting at, here. If I'm understanding right, you're saying that some of Psychology's most fundamental theories may be incorrect due to questionable methodology? If so, I'll say that the "bedrock" stuff has likely been retested in various forms all throughout the world and over an exteneded period of time. While I do think that some research may be adveresly affected by poorly representative participant samples, I'm confident that the field has a good handle on the "bedrock" stuff.

What's interesting is that replications don't get published that much, unless they happen to be failures to replicate on important issues. Does that mean replications have been attempted a lot more often, and researchers don't attempt to publish (or editors decline to publish) the ones that do succeed? Or are replications just not attempted that much in psychology? The interesting thing about this group of replications is that they ended up publishing all the "registered" entries, whether they failed or succeeded - and there was a huge amount of failures in that group. The concern is that there are many, many more unreplicable studies/concepts sitting out there, accepted as true, that simply haven't been put to the test.

There are perverse incentives in any peer-reviewed science to attack the things you can, and leave alone replications of other studies you don't expect to fail. And that professor sounded really, REALLY whiny about the process.

@486,739 (Killer Lettuce !!iNo3FkiZx)

> Well... Okay. You seem like you're being sincere, so I'll post my thoughts.
>
> Yeah, I have no problem admitting that it is an issue in the field. When evaluating the sources I have gathered for an essay, I find that I can often put in "low population validity" (meaning that it is not representative) as a criticism. Here, I found an interesting article that explains the problem quite clearly and concisely. Basically, it happens because college students are convinient, and researchers and journals tolerate it.

Yes, it's convenient - and really fucking lazy. Lots of academics in other fields look down on research psychologists for doing this. It shouldn't be tolerated at all. Pilot studies, maybe... but there has got to be an effort to make these results stick in other settings.

Kind of funny that the one study discussed above was based on one group of college students, and the failure to replicate was based on one group of college students in another country. The original author complained, in part, that you can't expect to replicate results in one country in another. HELLO?! What have you just admitted about the generalizability of your entire research program?!

@486,742 (cccuuunnttt)
> Does that mean replications have been attempted a lot more often, and researchers don't attempt to publish (or editors decline to publish) the ones that do succeed? Or are replications just not attempted that much in psychology?
I don't think so. There's plenty of research replicating, building upon and modifying previous research in order to better examine a theory. Just like any field of study.

@previous (cccuuunnttt)
> Yes, it's convenient - and really fucking lazy.
I get the feeling that you thought that I was defending it in some way? I wasn't, is indeed a problematic methodological flaw that can lower a reserach's generalisability.

@previous (Killer Lettuce !!iNo3FkiZx)
No, didn't think you were defending it. As far as the replications--to the extent they are also done on small samples of college students, not sure how much they help, I guess.

@OP
> Marcus pls respond too
Super. Wall o' text comin' up. Spoiler: Gin may be partly responsible for some or all of this post.

> This is one of my biggest problems with psychology as a whole
First, you seem to be conflating the entire field of psychology with social psychology in particular. Biological psychology, as the name implies, overlaps heavily with biology, and it is mainly focused on figuring out how all the different biological parts work together to contribute something to what eventually becomes a mental state, or a feeling, or an idea. How does damage to an area or disease effect a person's thinking or mental state? With tools like fMRI, PET scans, electro- and magneto-encephalography, and a host of other rapidly growing technologies, neuroscience explores which brain areas and neural tracts are active during different tasks and in response to different stimuli in real time. Cognitive psychology -thinking about how we think- has a number of different areas that work by hypothesizing mathematical models and testing whether they match human responses. Sensation and perception studies (and measuring people's response to them: A rocking little field called psychophysics) have values that have been measured over and over for almost a century in some cases. (It's not that cool, it's really just asking about the limits of human perception... but "psychophysics" is fun to say) Studies on learning and memory have similar results that have been replicated so much they are the "baseline" for every current-day study. So, we're not all just handing out a survey to twenty people in our class and getting excited if we see p ≤ 0.05 in our SPSS results.

And not to knock social psych... It's asking some interesting questions. (As a field, that is. Personally I don't give a fuck if cleanliness is related to moral judgments as seen in the OP. I'm not a social psychologist, but I might find that an interesting question if I were.) Social psych. also uses pretty mathematically sound -though not without some good criticism- methods like factor analysis to get to big ideas. (e.g. the "Big 5" personality measures) Again, it's not like the methodology is perfect, but whether it is sound is actively being debated.

> it uses tiny and unrepresentative samples, usually college students enrolled in a psychology class.
This is very much a valid criticism. (of those kinds of studies) Okay, if you thought the above was a defense of psychology, then lets get into the bad stuff. In the past psychology has often been derisively referred to as "the study of the upper-middle class white kid" or something similar because of its tendency to use students as convenient test subjects - and it does. Well, student projects do. Some studies do too - some studies do it due to constraints of time or funding, some studies do it as a means to get published, some studies do it because they're just being sloppy. Things that are easily generalizable (base human reaction times, neural correlates to a particular kind of stimuli, brain structure, and things that have actually been found not to vary across widely different populations use small sample sizes with little worry) tend to do this, but they do it with the knowledge that the small sample sizes are enough to generalize to a wider population.

So... The problem of sampling. We can't ask everyone in the world and then track them like Big Brother to form a coherent idea of how everyone will react all the time, so we have to make generalizations. And these generalizations have to start somewhere. We will never be able to get a perfect random sample of every human. We have to make some compromises. When and where it is feasible, the good researchers try to make their samples as random and representative as possible, even doing tests to control for various factors that might interfere with the conclusion they are trying to reach. It's not always feasible. It's not always worth it. If you are merely suggesting some relation between two measurable things as brain-food for anyone looking to advance or disprove a theory that involves those two things, then it might be worth publishing your humble results on the grounds that someone might get an idea to take it a step higher. Not every study is "Capital T" Truth. Trial and error is how science works. People who do science often realize this but get annoyed with the general public when they think something is true because a new study just came out.

> I've been reading some commentary from observers in other fields, and one of the most common refrains is how shocking it is that some of these studies get published in the first place.
Well that's nice. For chemists one carbon atom is exactly like another. For engineers, they are dealing with a specific problem of getting a bridge to span x meters or whatever. We can't treat people like carbon atoms, and we can't treat our theory like getting the flexural stress of steel bolts within the particular bridge's weight limits. We can't consult The Handbook of Chemistry and Physics to figure out how human subjects will react to such and such pressures and social variables within some social context. Some of us are actually working to estimate those values in relevant situations - because, you know, a fucking manual doesn't exist yet, but we can try to help with that by making observations - There are a million variables though, and it's often difficult to see how they interact. Everything in making assumptions about people in groups is context dependent. For every "x may effect y" you're really saying "x may effect y in a certain population under certain circumstances"; whether those circumstances are likely to change within the next few years or are relevant to other people somewhere else is an open question. It's not an exact science. It's a bit a mess. (At least the social part of it is. :/)

There are definite ideas about how large a sample size has to be to be statistically significant, and these are things that organizations taking opinion polls or testing vaccines pay close attention to - and those numbers are often too large for your average social psychology study to fund. Many good researchers will use preliminary results on a small sample size to question whether a larger theory is true or false or use the results as a springboard for further research. Yes, a lot of crazy shit gets published that should have pruned in the peer-review process (the current state of academic publishing as a money-making engine is a whole other issue that needs to be fucking remedied BTW), but the nature of science is to throw ideas out there and see what sticks. Again, not every "scientific" study is absolute truth - you and everyone else are free to critique it. Trial and error and discussion about how something succeeded or failed is a part of the process. (And really, don't get me started on click-bait pop-sci articles that claim some new thing has been "discovered" when it only references some insignificant study that sort-of pointed to evidence of this if some outlandish theory proved correct. This is actually hurting people's perception of science.)

> I realize that sample size is not everything, but do you have any qualms about so many of your field's bedrock principles being based on questionable studies
What are these "bedrock" principles and the questionable studies they are based on? Psychology barely has any principles that could be called "bedrock" last I checked unless you want to question whether we think with our brains or question that nerves send signals or that we are conscious... Actually Dan Dennett really did question whether we are conscious, and many people in neuroscience laughed nervously and cheered him on -and not because he was necessarily wrong about or how he was approaching it.

Fuck. I also wanted to bring up the meta-analysis as well, which takes data from numerous studies and analyzes those data together. This is actually pretty common, particularly when closely debated theories are at stake. So saying that there aren't people trying to find a more representative sample size or find a more generalizable result is a little misinformed.

(Edited 1 minute later.)

@previous (C)
I don't find meta-analyses that helpful, actually, if they are still based on small, unrepresentative samples. The underlying data is still questionable.

Marcus thank you for coming in and properly defending psychology

your post was very educational

@previous (cccuuunnttt)
Keep in mind, they can draw from all over the world and from all sorts of demographics. Whilst many might still have the college student issue, they aren't as hobbled as single studies by that issue. Additionally, not all examining theories that are affected by this to begin with.

@487,161 (cccuuunnttt)
> I don't find meta-analyses that helpful, actually, if they are still based on small, unrepresentative samples.
Then I don't think you actually understand what a meta-analysis is. There a number of ways to do meta-analyses. If the studies are actually trying for replication of results as (all or part of their goal) it may be possible to combine datasets. A number of studies have done this. Statistically they don't still suffer from the original problem of smaller datasets. (There are, of course assumptions here)

Spoiler: Doing a meta-analysis is hard and you have to think about the numbers you are crunching. Some are done well; some are not. If you have any specific criticism of any specific meta-analyses you wish to critique, then I would be happy to give you my feedback

@487,164 (B)
It seems like Becky just has some knee-jerk response and hatred of psychology. Doesn't it? Doesn't the very existence of things like meta-analyses or measures of effect size sort of throw some cold water on her parade of crazy?

Seriously KL:
She doesn't know the field.
She doesn't read the journals.
She doesn't see the debates about theories or actually know the theories being debated.
She doesn't keep up with the literature. (Which includes people debating ideas and debates about how to prove them)
She basically has decided that people debating about stuff don't know stuff.

Thanks Becky. You obviously don't know shit about any of this shit, but you are willing to say that people who have been looking at all of this shit for a lifetime know about as much as you. I think you will do well as a lawyer.

Keep arguing bullshit, and I hope you pass the bar. I think the ability to pretend to know more than other people "studying" this kind of thing and act all skeptical is a plus. Good luck. It looks like you are well on your way to being --annoying-- a lawyer.

EDIT: No I really do, Becky. Best of luck. Keep to it. Stop crying. Your're a golden goddess.

(Edited 8 minutes later.)

@487,309 (C)
I am always skeptical about the study of the studies if you want to be sure what they add up to. So tell me about confidence intearval. x,y and z when I look at the studies. So of course I hit net and discover confusion can go meta. I may be prejudiced by Meta studies as they most Confucius me. As with Statisticians in general if paid to prove a point with math then I C fuzzy math applied.

I use fuzzy math/logic in my work but wood never use it to prove a point on a forum. For a client depends on how deep their pockets are.

@previous (Syntax)
> I am always skeptical about the study of the studies
Yes. And other studies are too. That's what fucking science is about. It's a conversation about whether x causes/is associated with/doesn't depend on y. Skepticism is a virtue.

> So tell me about confidence intearval. x,y and z when I look at the studies.
Most do. It's not that hard to find. Giving a lecture or presentation in front of a group without showing something as basic as confidence intervals isn't done. (At least not when I was in grad. school, and certainly not in something that got published.)

> I use fuzzy math/logic in my work but wood never use it to prove a point on a forum. For a client depends on how deep their pockets are.
I lost my decoder ring. Are you talking about fuzzy logic? There's a lot of good work going into that in regards to control systems. (maybe satellites, I don't know, It works for subway cars. I can't imagine satellites going into "good enough" positions...) Fuzzy math doesn't really make sense unless you are just fudging results and hoping they're "close enough" to be correct.

(Edited 2 minutes later.)

@previous (C)
It wood take a book to explain where the fuzzy math is used in multiple control systems of Satellites.
> Fuzzy math doesn't really make sense unless you are just fudging results and hoping they're "close enough" to be correct.

Each company that designs them has closely guarded secrets about how they use fuzzy math and yes its a close enough to be correct issue. Think of it this way. Usually lots of feedback loops are necessary in control systems. Its the very essence of feedback/control. Yet in space where cosmic blasts of gamma rays - X-rays and other nasty particles hit IC's a momentary glitch can break the loop and valves can open and gas can be expelled or ion engines that keep GEO's in position can mess up, but usually its the fly wheels of several types that suddenly jerk and cause the Satellite to move in a jerky manner. Gyros on tight controls and yes they are in feedback loops, but when sensing Cosmic fuck ups of space the loop is quickly broken to a Non loop mode and a fuzzy loop is switched in for as long as needed.

> Skepticism is a virtue.

Usually I encounter proofs in technical papers that end up using Meta values to prove a point and I am always very skeptical because a quick look with Google and papers published so often disagree from one paper to the other and its often the Meta data that pulls it one way or the other. Stats in school were always my weak point - I was great with the math but trusting my results never pleased me.

I could fudge that math to change my results and almost prove the opposite position - Clients often ask me to do this - on occasion I even admit to playing my math to please a client but am careful to add a bottom line where I say I can prove it differently and then the client is free to remove my bottom line for presentation - AT&T who I abhor does this to my work product all the tyme. I have even cited Meta Analysis to prove a point but only under duress.

So with ultra critical control systems in space instead of a tight or even soft feedback system one uses no loop at all. Instead its open and fuzzy with boundary sensors that sense a drift to out of bound limits and then send a signal to the control to send a form of command that slowly moves the system back in the other direction - Kinda like a hunt and peck - Its drifting between two points and is very fuzzy in between the boundaries. AND YES its always drifting and fuzzy yet safe from a Cosmic blast from Sun flares or even black holes which pump shit into space on regular basis.

@previous (Syntax)
> It wood take a book to explain where the fuzzy math is used in multiple control systems of Satellites.
Great. I just Googled, and fuzzy math is sort of a synonym for fuzzy logic and fuzzy set theory, the latter of which is new to me! Fuzzy set theory makes some sense though given the problems of traditional set theory, so let's just take fuzzy sets and run with them. I like my sets fuzzy anyway.

> Usually lots of feedback loops are necessary in control systems. Its the very essence of feedback/control.
Yes, the fuzzy logic part usually comes in handy in making adjustments based on input of some intensity, so it makes sense. Again, these kinds of control systems work to control subway braking systems without throwing passengers about like crazy. I can imagine it would be useful in other fields.

> Yet in space where cosmic blasts of gamma rays - X-rays and other nasty particles hit IC's a momentary glitch can break the loop and valves can open and gas can be expelled or ion engines that keep GEO's in position can mess up, but usually its the fly wheels of several types that suddenly jerk and cause the Satellite to move in a jerky manner. Gyros on tight controls and yes they are in feedback loops, but when sensing Cosmic fuck ups of space the loop is quickly broken to a Non loop mode and a fuzzy loop is switched in for as long as needed.
Yeah, here I don't see how it would be done. I can see fuzzy control systems bringing a satellite into orbit based on gyroscope feedback or something, but sudden solar flare activity would just be random noise that still may be too intense to deal with. Anyway, I don't know. I'm not a rocket surgeon.

> >Skepticism is a virtue.
> Usually I encounter proofs in technical papers that end up using Meta values to prove a point and I am always very skeptical because a quick look with Google and papers published so often disagree from one paper to the other and its often the Meta data that pulls it one way or the other. Stats in school were always my weak point - I was great with the math but trusting my results never pleased me.

I don't know know what you're trying to say here, but I think that you mean that trusting in your own results without independent verification doesn't make you happy... And it shouldn't! You should always be skeptical of everything! Even you're own results! Don't trust anything -not even your own findings- until someone else comes to the same conclusion independently!

> So with ultra critical control systems in space instead of a tight or even soft feedback system one uses no loop at all. Instead its open and fuzzy with boundary sensors that sense a drift to out of bound limits and then send a signal to the control to send a form of command that slowly moves the system back in the other direction - Kinda like a hunt and peck - Its drifting between two points and is very fuzzy in between the boundaries. AND YES its always drifting and fuzzy yet safe from a Cosmic blast from Sun flares or even black holes which pump shit into space on regular basis.

I'm adding this to the Syntax Markov corpus.

(Edited 2 minutes later.)

@487,310 (C)
I think that sometimes it's trolling. But at times she comes across like one of those fundie types who try to disprove any and all science that disagrees with their worldview. It's unsettling.

Hey, I'm curious, whereabouts are you in psychology, education or career-wise? You seem to really know your stuff.

@previous (Killer Lettuce !!iNo3FkiZx)
I was once-upon-a-time in a Cognitive Science Ph.D. program. I was researching mathematical models of vision and depth-perception-from-motion. I went through undergrad with a major in psych. and a rabid interest in computer science and graphics programming.

I'm a little tipsy right now. If I seem a little angry at Becky... I'm really not. (sorry Beckers) Most of my friends and I argue back and forth relentlessly and scare people because they think we're really angry or about to come to blows or something. We're really not. We just like arguing with one another. We do the same thing when we play games.