Ways of Knowing

 

- Above by Charles Pierce - a 19th century mathematician

- Discuss these ways of knowing – what are the strengths and weaknesses of them.

 

 

Ways to Make Informed Decisions

 

 

Notes:

 

Say you want to make a decision--

• ex.- adopt a new method of teaching reading,

• modify the behavior of a depressed adolescent

• try a new articulation therapy

• use a new/better teacher evaluation form that will result in improved

instruction, etc.

What could we base this decision on?

need some info to be able to make a sound, informed decision

1. personal experience-

can be biased by personal feelings,

may not have knowledge- decision is beyond scope of experience,

could be quick & easy way to decide, but memory is faulty and subject to

errors in recall

2. expert authority-

even authorities are not always right,

also dealing with someone else's personal experiences, also biased,

authorities don't agree on everything

3. search the literature-

• much work, time consuming,

• can review past research of all authorities and get broader perspective

than from just one

4. do research-

• also much work, time consuming

• BUT can sometimes get a specific answer to a specific local problem by

doing action research

difference between library research and RESEARCH

 

The Scientific Method

1. identification of problem

2. form a hypothesis

3. deductive reasoning- decide on procedure

4. data collection and analysis

5. derive conclusion

Note: We never prove a hypothesis -- confirm or fail to confirm

Notes:

approach to acquiring knowledge in the natural sciences

  1. identification of problem
  2. hypothesis
  3. deductive reasoning- decide on procedure
  1. data collection and analysis
  2. derive conclusion

never prove a hypothesis-- confirm or fail to confirm

explains observed phenomenon [explain why]

should be consistent with previously established knowledge

should be verifiable [can we test it?]

should stimulate further research

put in science-practitioner model

 

Lesson 2

The Scientific Method

1. identification of problem

2. form a hypothesis

3. deductive reasoning- decide on procedure

4. data collection and analysis

5. derive conclusion

Note: We never prove a hypothesis -- confirm or fail to confirm

 

Stages in the Research Process

Select problem area

Derive hypothesis(es)

Review the literature

Develop methodology

Data collection

Data analysis

Interpretation of results

Select a Problem or Research Topic

    1. contributes to the professional bases.
    2. something that will stimulate and motivate you.
    1. your interests
    2. building on previous research
    3. the role of theory
    4. the utility of your research (meaningful)

Questions and Hypotheses

Question: Questions about the relationship between or among constructs.

(constructs): "an informed, scientific idea "constructed" to describe or explain a behavior" e.g., intelligence or anxiety

Hypothesis: States the expected relationship between constructs.

    1. lack of relevant and extensive knowledge about a topic.
    2. can’t measure and test constructs
    3. unable or unwilling to be specific enough

General Types of Questions or Hypotheses

      1. asks a question about
      2. the relationship between 2 or more constructs that
      3. can be measured and tested
      1. the topic
      2. specific constructs of interest

Types of Hypotheses

Non-directional hypothesis

Directional hypothesis

Null hypothesis

Notes:

 

 

 

Practice on hypotheses

  1. Will abused children who attend play therapy sessions for 6 months do better on measures of appropriate classroom behavior than children who attend 6 months of non-play therapy?

 

RESEARCH QUESTION

 

  1. At the end of 6 months, there will be a difference in appropriate classroom behaviors between those children who attend play therapy sessions and those who attend non-play therapy.

NON-DIRECTIONAL

  1. At the end of 6 months, there will be no difference in appropriate classroom behaviors between those children who attend play therapy sessions and those who attend non-play therapy.

NULL

  1. At the end of 6 months, there will be children who attend play therapy sessions will do better on measures of appropriate classroom behavior than children who attend non-play therapy.

DIRECTIONAL

Selecting a Hypothesis

interest - what are you dying to know?

feasibility - is research problem "do-able"?

ethics - is research ethically tenable?

Where to get ideas for hypotheses

personal experience

textbooks

library books

journals

Operational Definitions

Variables & Levels

 

- population of interest – what group we are interested in making generalizations about

 

Independent and Dependent Variables

1. A survey of 1712 high school seniors in 421 high schools across the country revealed that 74% percent of students think that their teachers are doing a good or excellent job. Twenty-two percent of those surveyed said that they would like to become teachers. Although the seniors reported that their teachers generally have good content knowledge and are competent, the seniors also said that many of their teachers were not as interesting or creative as they should be.

Population of interest:

Independent variable:

Levels:

Dependent variable:

2. This study investigated whether absent students whose homes received phone calls via a computer-activated message device had a better school attendance record than students whose homes were not called. 150 students in three high schools in Pirogue Parish were randomly selected to receive the messages whenever they were absent. 150 students randomly selected students served as the control group. They did not receive any absence notification. After 60 days, the attendance records for the groups were compared. Results revealed that students whose homes were called were absent significantly less than students whose homes were not called.

Population of interest:

Independent variable:

Levels:

Dependent variable:

3. The researcher hypothesized that peer evaluation as part of the writing

process would lead to improved attitudes toward writing as measured by the

Writing Attitude Scale for Students (WASS). Four intact classes of eighth

graders were randomly assigned to the treatment group. This group received peer evaluation training and utilized peer evaluation during three writing

assignments. Four randomly selected classes served as the control group. They received feedback from teachers only after their writing assignments. Both groups completed the WASS after the research period was over. Results indicate that the treatment group had significantly higher scores on the WASS than the control group.

Population of interest:

Independent variable:

Levels:

Dependent variable:

4. Children who entered kindergarten at age 5 were compared with children who entered kindergarten at age 6 on measures of academic achievement taken at grade 5. Results indicate that children who entered kindergarten at age 6 scored significantly higher on standardized tests measuring reading achievement and mathematics achievement.

Population of interest:

Independent variable:

Levels:

Dependent variable:

5. This study explored the relationships between measures of computer science aptitude, mathematics achievement, and writing achievement. For the 142 high school students in the sample, it was found that there is a moderately strong pattern of relationship (+.71) between computer science aptitude and mathematics achievement. However, there was no relationship between computer science aptitude and writing achievement.

Population of interest:

Independent variable:

Levels:

Dependent variable:

6. The effects of social skill training on a 29 year-old mentally retarded male adult were explored. The subject listened to typical social situations (such as getting a compliment or saying thank you) on audiotape and discussed the situations with a therapist. The subject's positive social verbal interactions were counted before the training, at three times during the training period, and at three times after the training had been completed. All counts were taken by observing the subject at a evening recreation time in the subject's group home. Results show that positive interactions increased during the training period but then rapidly decreased after training had stopped.

Population of interest:

Independent variable:

Levels:

Dependent variable:

7. A researcher wants to investigate the changing roles of working mothers and the pressures they and their children face. The researcher observes the

behaviors of 12 four year-old children in a day care setting for 6 to 8 hours per week for 10 months. The mothers are observed as they drop off and pick up their children. The mothers and the day care workers are interviewed. The researcher discovers several recurrent themes in the observations and interviews.

Population of interest:

Independent variable:

Levels:

Dependent variable:

 

Notes:

1. High school students

None --Trick question! Descriptive research doesn’t have IVs and DVs

2. high school students in Pirogue Parish

computer-activated phone calls

got messages vs. did not get messages

attendance

3. eighth graders

peer evaluation

peer evaluation vs. teacher feedback

attitudes toward writing

4. school children

age of kindergarten entrance

age 5 vs. age 6

reading and math achievement

NOTE: cannot infer causality

5.high school students

None - Trick question

Correlational studies do not have independent and dependent variables

6. No population - single subject

social skill training

no training (before) vs audiotape/therapist (after)

social skills

7. working mothers, day care workers and preschoolers

None- ethnographic studies don’t have IVs and DVs

 

 

Parts of a Research Article

ABSTRACT

INTRODUCTION

REVIEW OF THE RESEARCH

HYPOTHESES OR OBJECTIVES

Notes:

The organization of primary research articles follows the steps in the

scientific method

abstract --brief overview of the article-- usually 200-250 words maximum--

convenience to reader, not all journals require an abstract

1. introduction

states the problem in a general way.

cites important previous theory.

justify the importance of the study-- importance should be objectively clear

2. review of research

cite previous research -- what is the background in the field that leads to

your study?

should be evident where your research fits

--look for evidence of bias in prior research

• who is the author?

• what is the author's affiliation? does affiliation indicate bias

• if author is strong proponent or opponent of certain theory, may be an

indication of bias

does author cite relevant research?

• usually key studies will be mentioned over and over again, if these are

missing, may signify that author hasn't done a thorough review

• is review of research biased toward a particular viewpoint?

• are contradictory studies ignored?

•is biased language used?

--how many articles in a review of research?

•can't review everything, depends on journal space; general guideline, 5-10

key articles should be cited, if only briefly for articles - more in thesis

3. hypothesis -- research hypothesis is a statement of what we expect--

we make a guess about the relationships between variables or the differences

between two treatments, etc.

-may be a statement or in question form

-a good research hypothesis: 1. sets up a "testable" situation 2. gives

direction to research 3. identifies the variables of importance 4. is

grounded in theory 5. is brief but with clarity

Some studies use objectives: instead of a hypothesis: descriptive study,

ethnography

ex.- do descriptive study of teacher salary -- look at salary schedules and

policies

objectives are to describe level of salary for state and education levels,

ex.- study sex-role related prejudices in kindergartners

observe sex-role related play, record instances of peer learning of sex-role

related behaviors, look at influence of teacher

 

METHODS

METHODS

SAMPLING

INSTRUMENTATION

RESEARCH DESIGN

DATA ANALYSIS METHODS

RESULTS

DISCUSSION/IMPLICATIONS

 

Notes:

 

4. methods

sampling -- how was sample selected?

what does sample look like?

can't study entire population

want to get a sample that reflects the population

data collection - what data was collected

how was data collected

does data seem to be reliable and valid (Construct vs. Indicator)

statistical analysis

how was data analyzed?

5. results

data crunching results are given with level of statistical significance

6. conclusions

are conclusions warranted? or do they go beyond the results?

Look at article critique --website: Author makes statements about teaching

effectiveness, not warranted by what was investigated

do conclusions answer the research question?

do conclusions agree with previous research?

what is the future of research in this field?,

good research often generates more questions than you answer

 

 

 

Choosing a Design

    1. Randomization
    2. Building conditions or factors into the design as independent variables (e.g., degree of depression – could categorize as high or moderate and randomly assign from those groups [differences within groups would be randomly distributed])
    3. Holding conditions or factors constant (e.g., time of day, therapist) – takes out the influence of those variables but may restrict the generalizability
    4. Statistical adjustments – removes the effect of the control variable (pretest scores controlled)

  1. Freedom from bias – the data and the stats computed from the data do not vary in some systematic way. Any differences could be then attributed to the independent variables. (e.g., if testing two counseling treatments and one group was severely depressed while the other group was slightly depressed bias would exist – could not compare the two treatments)

  1. Freedom from confounding – bias can be introduced by the confounding of variables. 2 or more variables are confounded when their effects can’t be separated (e.g., two treatment methods with two different therapists – does treatment or therapist have an effect?)
  2. Statistical precision for testing hypotheses – the data must be adequate to accurately test the hypothesis. This is achieved through proper design and good measurement.
    1. make levels of the IVs as different as possible (e.g., if doing treatment and you do all treatment groups, similarities may exist between treatments)
    1. Error refers to any event, characteristic, or situation that is unsystematic and causes measurements to fluctuate randomly
    2. Standardization of treatment and measurement procedures helps to reduce error.
    3. Pilot research also helps smooth out problems with research that can cause error
    1. Extraneous variables cause systematic error or bias. While error variance is random (affects subjects or measurements in no particular order), error due to extraneous variables causes measurements to be pulled in a similar direction.

 

 

  1. Find out what is already known about the research topic – what can you add?
  2. Find out how other people have made designs to explore the research question and see if you can add to the research by doing it differently
  3. Consider the resources available to you
  4. Consider the threats to validity
  5. Find the best match of 1-4

 

 

 

 

 

Experimental Designs

  1. Adequate experimental control – there must be enough constraints on the conditions of the experiment so that the researcher can interpret the results (e.g., if you are testing two types of treatments with adolescents, can you control account for other variables that may confound the findings [treatment provider, setting] and isolate the effects of the experimental variable [treatment method])
  2. Lack of artificiality – especially important in counseling research. Can the findings be extended to the "real world."
  3. Basis for comparison – must be able to make some comparison to determine whether there is an experimental effect (control, comparison, or external group) e.g., treatment method with no comparison – is the treatment better, worse, the same as no treatment?
  4. Adequate information – data must be adequate for testing the hypotheses of the experiment
  5. Uncontaminated data – data should be well measured and represent the experimental effects
  6. No confounding variables – should be no other variables having an effect on the dependent or variable. If so, these variables should be separated or controlled through the experimental design or statistical manipulation.
  7. Representativeness – results should be able to be generalized beyond the sample (except in action research). Usually done with random selection or assignment.
  8. Parsimony – all other things being equal, a simpler design is preferred.

 

 

LESSON 3

Design Validity

 

 

 

 

Relationship between internal and external validity

 

 

 

 

 

 

Gelso's Research Classification

 

Notes cont. ---

    1. Power – power is the probability of correctly deciding that there is a relationship if one truly exists (e.g., probability of deciding that counseling method has an effect on depression if it really does)

State of Nature

Decision Null is true Null is false

Reject null

Type I error (alpha)

Correct decision

Do not reject null

Correct decision

Type II error (beta)

power = 1 - beta

    1. Violated assumptions – statistical tests have assumptions. If the assumptions are not met, we may come to inaccurate conclusions.
    2. Fishing and error rate – finding significant results due to chance
    3. Unreliability of measures
    4. Unreliability of treatment implementation – for example if therapist in our example use different treatment modes, the true relationship between method and depression may be obscured
    5. Random irrelevancies in the experimental setting – anything that may lead to differences in how experimentee’s respond (excluding treatment) e.g., in our methods example, if several members of the experiment also go to a support group for depression outside of the experiment.
    6. Random heterogeneity of respondents – differences in subjects that may lead to differences in their responses (e.g., some of our participants may have many resources and may be more likely to respond to treatment).
    1. History – events that occurred in addition to the treatment (e.g., TV show on depression aired during the experimental period
    2. Maturation – things that normally occur over time (growing older, hungrier, tired)
    3. Testing – taking a test more than once (pre and post) - does the initial testing influence the outcome - act as a treatment? most problematic if testing done on single group.
    4. Instrumentation – changes over the course of the experiment in testing instruments or observers (e.g., if checking behavior of children over time, observer may become more observant or less attentive) - also applies to diffent processes or procedures (two therapist doing different groups)
    5. Statistical regression – measurements tend to go toward the mean over time (real low scorers will tend to score higher and high scorers will tend to score lower – because other factors [ anxiety, depression] may have influenced scores in the first place)
    1. Selection – particularly important when we do not have random assignment to groups
    2. Mortality
    3. Interactions with selection – even in cases of random assignment to groups interactions can occur. For example in our group getting the treatment is from Liverpool and the other group is from Oswego. Group 1 did not have an opportunity to see the TV show on depression while the Oswego group did. Selection X History
    4. Ambiguity about the direction of causal inferences – especially important non-experimental designs – correlation does not mean causation.
    5. Diffusion or imitation of treatments – effects from treatment groups spread to non-treatment groups (e.g., our treatment group talks to the non-treatment group about what they are doing).
    6. Compensatory equalization – those not getting treatments get some type of treatment elsewhere
    7. Compensatory Rivalry by subjects receiving less-desirable treatments – desire to outperform the treatment group
    8. Resentful demoralization of subjects receiving less-desirable treatments – opposite of #12 e.g., in our study the non-treatment group may become more depressed because they are not getting treatment
    1. Inadequate preoperational explication of constructs – not well defined or operationalized (e.g., play therapy)
    2. Mono-Operation bias – using only one measure of some construct with the DVs and IVs. May not capture the whole essence of the constructs being defined.
    3. Mono-method bias – using the same method of measurement for multiple DVs or IVs. e.g., using two self reported measures of depression rather than a self-report and an family member evaluation
    4. Hypothesis guessing within the experimental conditions – when people guess what the experiment is and comply or rebel. Give example of study with Joe’s depression study.
    5. Evaluation Apprehension – people may respond in different ways because of being evaluated (may want to look good)
    6. Experimenter Expectancies (the experimenter responds differentially in treatment or non-treatment groups)
    7. Confounding Constructs & Levels of Constructs – occurs when only segments of a scale are used e.g., if we want to examine the relationship between severity of depression and effectiveness of a treatment, but only select people on the high end of depression severity, we may not see the true relationship
    8. Interaction of different Treatments – when one treatment follows another. You can’t determine whether observed changes were due to one of the treatments or the interaction of the treatments.
    9. Interaction of Testing and Treatment – treatment and pretest combine to form an effect e.g., pretest sensitizes people.
    10. Restricted Generalizability across Constructs – we can only make generalizations about the specific constructs used in the experiment. Often we fail to look at more important constructs.
    1. Interaction of Selection & Treatment – To what extent can results be generalized to different types of people (gender, intelligence, race, …). Generalizations can only be made to groups that were selected for the study. Better to use subjects from different groups e.g., Holland’s initial vocational development research.
    2. Interaction of Setting & Treatment – To what extent can results be generalized to different settings. Generalizations can only be made to those settings that were used for the study. Better to test at different settings.
    3. Interaction of History & Treatment -- To what extent can results be generalized to different time periods. We can enhance validity across time by repeating research at different times.

 

 

 

 

 

Validity Practice

Mary has designed and run an experiment to determine the effects of counselor well-being on client outcome. She hypothesized that counselors who were emotionally healthy would have a more positive effect on their clients (i.e., clients would do better in therapy) than counselors who were emotionally unhealthy.

Mary decided to measure counselor emotional state with a measure of trait anxiety; the Counselor-Reported Anxiety Profile. Client outcomes would be measured with a 20 item questionnaire designed to measure the clients self-reported benefits of therapy. The client self-report measured client satisfaction with their counselor, client perception of change in therapy, and client perception of their counselors involvement in therapy.

Mary recruited subjects for her study from a women's center where she worked part-time. Therapists for the study were recruited from the counseling master's program at the local university. Four counselors volunteered for the experiment and were accepted. The counselors were tested to with the Counselor-Reported Anxiety Profile. All were above average on the scale, indicating that all were experiencing higher than normal levels of anxiety. Each counselor was randomly assigned to a counseling client and therapy was conducted for 6 weeks (one 50 minute session per week). At the end of the 6th session, each client was asked to fill out the questionnaire.

Mary matched the client scores with the counselors' test results and did a correlational analysis. She did not find a statistically significant correlation and concluded that counselor emotional health was not related to therapy outcome.

    1. History – events that occurred in addition to the treatment (e.g., TV show on depression aired during the experimental period
    2. Maturation – things that normally occur over time (growing older, hungrier, tired)
    3. Selection – particularly important when we do not have random assignment to groups
    4. Ambiguity about the direction of causal inferences – especially important non-experimental designs – correlation does not mean causation.
    5. Diffusion or imitation of treatments – effects from treatment groups spread to non-treatment groups (e.g., our treatment group talks to the non-treatment group about what they are doing).
    1. Inadequate preoperational explication of constructs – not well defined or operationalized (e.g., play therapy)
    2. Mono-Operation bias – using only one measure of some construct with the DVs and IVs. May not capture the whole essence of the constructs being defined.
    3. Evaluation Apprehension – people may respond in different ways because of being evaluated (may want to look good)
    4. Confounding Constructs & Levels of Constructs – occurs when only segments of a scale are used e.g., if we want to examine the relationship between severity of depression and effectiveness of a treatment, but only select people on the high end of depression severity, we may not see the true relationship
    5. Interaction of different Treatments – when one treatment follows another. You can’t determine whether observed changes were due to one of the treatments or the interaction of the treatments.
    6. Restricted Generalizability across Constructs – we can only make generalizations about the specific constructs used in the experiment. Often we fail to look at more important constructs.
    1. Interaction of Selection & Treatment – To what extent can results be generalized to different types of people (gender, intelligence, race, …). Generalizations can only be made to groups that were selected for the study. Better to use subjects from different groups e.g., Holland’s initial vocational development research.
    2. Interaction of Setting & Treatment – To what extent can results be generalized to different settings. Generalizations can only be made to those settings that were used for the study. Better to test at different settings.
    3. Interaction of History & Treatment -- To what extent can results be generalized to different time periods. We can enhance validity across time by repeating research at different times.

 

Reliability

X = T + e

where X = an observed score

T = a true score free from any error

e = error

the observed variability of a set of scores is equal to the true variability plus error variance (random error)

so2 = st2 + se2

 

rxx’ = st2/so2

 

rxx’ = 1 - se2/so2

rxx’ is an estimate of the reliability of a set of scores

- there are methods of estimating reliability that are not bound to the test-takers but these methods are not frequently used (IRT methods)

    1. internal consistency
    2. stability
    3. alternate forms
    4. interrater reliability
      1. Cronbach's alpha
      2. Kuder Richardson 20 (KR20)
      3. Kuder Richardson 21 (KR21)
      4. Split-halves – correlate to halves of a test; there should be a positive correlation

**** do bullseye example *****

  1. test length – longer tests will be more reliable, but tests that are too long may cause test-takers to become fatigued or irritated – a good balance needs to be found
  2. homogeneity – As with any correlation, heterogeneity among a group increases variance and the resulting correlation. In reliability estimates, as group homogeneity increases, true score variance decreases and the resulting proportion of true score to observed score variances decreases (Crocker & Algina, 1986).
  3. item difficulty – related to homogeneity; if items are to difficult or too easy there will not be variance among the test-takers scores (e.g., everyone get a 100%)

Number of alternatives – the fewer the number of alternate answers – the better the reliability (e.g., t-f tests will tend to be more reliable than

Writing Research and Using APA

Sections of the Research Report

Title

Abstract

Introduction

Method

Subjects

Measures (or variables or instruments)

Materials

Design (or design and analysis)

Procedure

Results

Discussion (or conclusions)

References

Tables

Figures

      1. intro to the problem:
      2. development of the framework for the study:
      3. statement of the research hypothesis

 

The Research Problem

Introduction

Problem statement

Literature Review

Definitions

Specific research question

Significance of Research

 

Methods

Experimental Survey/Ex Post Facto Qualitative

Hypothesis Hypothesis Hypothesis

Subjects/Sampling Sampling Focusing

Treatment Instrumentation Sampling

Data Gathering Data Gathering Instrumentation

Data Analysis Schedule Iterations

Schedule Schedule

Pilot Studies

Human Subjects Concerns

Limitations

The Research Problem

  1. Introduction:
  1. Problem statement:
  1. Literature Review

      1. review relevant and recent literature related to the topic
      2. lead the reader to the question you are going to ask. The question should be obvious and inescapable by the end of the lit review.
  1. Definitions
  1. Specific research question
  2. Significance of Research

 

Methods

      1. Subjects
      2. Instrumentation
      3. Procedure
      4. limitations

Experimental Survey/Ex Post Facto Qualitative

Hypothesis Hypothesis Hypothesis

Subjects/Sampling Sampling Focusing

Instrumentation ** Instrumentation Sampling

Treatment Data Gathering Instrumentation

Data Gathering Schedule Iterations

Data Analysis Schedule

Schedule

** NOTE: I ADDED THIS

Pilot Studies

Human Subjects Concerns

Limitations

 

 

Boll, L. (1973). Effects of filial therapy on maternal perceptions of their mentally retarded children's social behavior (Doctoral dissertation, University of Oklahoma, 1973). Dissertation Abstracts International, 33(12-A), 6661.

Fall, M., Balvanz, J., Nelson, L., & Johnson, L. (1994). The relationship of a play therapy intervention to self-efficacy and classroom learning behaviors. Paper presented at the North Central Association for Counselor Education and Supervision, Milwaukee, WI.

Foley, J. M. Training future teachers as play therapists: An investigation of therapeutic outcome and orientation toward pupils. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED 067 794)

Wortman, P. (1994). Judging research quality. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis. (pp. 97-109). New York: Russel Sage Foundation.

Webb, N. B. (1991). Play therapy with children in crises. New York: Guilford Press.

Schaefer, C., & O’Connor, K. (1983). Handbook of play therapy. New York: Wiley.

Marans, S., Mayes, L., & Colanna, A. (1993). Psychoanalytic views of children’s play. In A. Solnit, D. Cohen, & P. Neubaur (Eds.), The many meanings of play. (pp. 9-28). New Haven, CT: Yale University Press.

CW4B

USING APA STYLE --

THE MOST COMMON MISTAKES

 

 

1. Inventing your own rules for format and reference lists.

APA is a very precise style. Check the manual for the rules which are

comprehensive. Look at the samples provided. Ask questions if you run across

anything which does not fit APA rules.

2. Using incorrect margins.

APA uses 1 inch margins on all sides. The bottom margin may be adjusted if

needed, for example to avoid putting a heading on the last line of the page or

to avoid putting the last few words of a paragraph at the top of the next page.

Page headers go inside the top margin.

3. Using an author's first name or using gender specific pronouns.

APA requires nonsexist language and references at all times. Proof your work

carefully. There is no reason to ever identify the gender of a researcher or

author. 4. Formatting the title page incorrectly. Look at the samples and the

manual to be sure. The page header and the running head are two different

things.

5. Not formatting the reference page correctly. Follow the manual closely.

The reference list is in alphabetical order; check the manual if there is a

question. The reference list is not a bibliography; there should be a

correspondence between the reference list and the references in the text.

6. Using quotations.

Avoid direct quotes unless absolutely necessary. Direct quotes are rare in APA.

Summarize and paraphrase instead.

7. Using incorrect spacing.

Double space the text. All text is flush left. Do not hyphenate words. Check

spacing between headings and text.

8. Incorrect capitalization in a book or article title.

Caps are used only for the first word, proper nouns, and the first word after a

colon.

9. Putting too much information in the reference list.

Use an issue number only when warranted. Give month, season, or exact date only

when warranted. Follow the manual.

10. Using first person.

Never refer to yourself (I, we, me, etc.). You may have to resort to passive

voice. Check the manual for hints on how to strengthen your writing style.

11. Using the wrong verb tenses.

Double check verb tenses. In general, when referring to research conducted in

the past, use past tense ("Smith concluded.."). When referring to the present

state of knowledge or theory, use present tense ("These studies suggest...").

When talking about your proposed methods, use future tense ("The data will be

collected..."). Look at the samples.

give our example of article critique

Ethical Issues

    1. nonmaleficence: "do no harm" -- in planning research you must be aware of harm or potential harm that may come to subjects
    1. beneficence: "do good for others"

    1. Autonomy

    1. Justice
    1. Fidelity: faithfulness

 

 

 

Issues in research ethics

As researchers we are expected to accurately report knowledge gained from research and to prevent the misuse of such knowledge

** discuss Lynne’s dissertation. does the publisher have the ethical right to refuse to publish based on fear of how that information will be determined? **

Publication credit

Subjects:

*** discuss forms and procedures **

    1. estimate the cost/benefit of the study -- do the benefits outweigh the costs; are the potential benefits greater than the potential harms?; who benefits, the client or the society
    1. minimize the risks -- are there other ways of designing research that would reduce potential harm?
    1. assess risks through pilot studies (with colleagues) or role-playing
    1. select subjects that are less likely to be harmed (e.g., don’t select severely depressed clients for participation in a study involving criticism from others)
    1. ALWAYS: consult with others

Informed consent:

** read section G.2. ***

Deception

1. determine if the potentials for harm

2. determine if other designs could be used

3. CONSULT with others

Confidentiality

Treatment issues

 

 

Experimental Designs

 

 

Types of true experimental designs

 

Posttest only control group design

Pretest posttest control group design

Solomon four group design

 

Posttest only control group design

 

RG1 X O1

RG2 -- O2

 

in a more general sense where there are K treatments:

 

RG1 X1 O1

RG2 X2 O2

. . .

. . .

. . .

RGk Xk Ok

RGk+1 -- Ok+1

 

 

Randomly Assigned Posttest

 

RG1 Group 1 15Ss getting RET (X1) O1 \

RG2 Group2 15Ss getting Behavior (X2) O2 - BDI

RG3 Group3 15Ss getting No Tx (-- ) O3 /

 

 

  1. controls threats to internal validity
  2. does not require pretest

 

 

  1. can detect differences between groups but not the amount of change
  2. threats to external validity

 

    1. interaction of selection an X: if treatment(s) are different – is it because of the particular sample or do results generalize?
    2. reactivity – are there treatment effects because someone is in an experiment?

 

Pretest posttest control group design

 

RG1 O1 X O2

RG2 O3 X O4

. . . .

. . . .

. . . .

RGk O2k-1 Xk O2k

RGk+1 O2k+1 X O2(k+1)

 

 

Advantages

  1. controls variance on DV and gives a more powerful test – if we just measure on the posttest (posttest only control group design) then the variance of the outcome measure would be composed of:

    1. error
    2. differences between treatment groups

 

if we use measure subjects at pretest we can define 3 sources of variance

    1. error
    2. differences between treatment groups
    3. differences between subject (reduces the error term)

 

*** do example on board of variance calculations **

 

  1. if subjects drop out of the experiment – we can see if people who dropped out were different in each group (e.g., did severely depressed subjects drop out of the treatment group but remain in the control group)
  2. pretests may be used to more accurately describe subjects in the study
  3. allows you to determine the amount of change from the treatments (and compare this to the amount of change in the control group)
  4. controls threats to internal validity

 

Weaknesses

 

  1. pretests may sensitize subjects to treatment – e.g., giving a depression inventory may make a treatment more effective because it causes the subjects to think about their disorder in a different way than if they had not been pretested. The treatment interacts with the pretest to make the treatment more effective than in the "real world" (threat to external validity)
  2. other threats to external validity

 

    1. interaction of selection an X: if treatment(s) are different – is it because of the particular sample or do results generalize?
    2. reactivity – are there treatment effects because someone is in an experiment?

Solomon four group design

 

 

 

RG1 O1 X O2

RG2 O3 -- O4

RG3 -- X O5

RG4 -- -- O6

 

 

Advantages

 

  1. controls threats to internal validity
  2. the researcher can detect if pretesting had an interaction with treatment (compare O2 to O5) – external validity threat
  3. since there are two experiments in one – it provides a natural replication – thus enhancing external validity

 

Weakness

 

  1. cost and time and number of subjects -- practicality

 

 

Factorial Designs

 

 

A counselor is interested in the effectiveness of two treatments with chronically anxious subjects. The treatments are progressive relaxation and systematic desensitization. He grouped the subjects into 3 groups – moderate, high, and extreme anxiety. It is possible that the 3 subject groups may respond differentially to treatment (anxiety level and treatment may interact) A 2X3 factorial design was used..

 

 

PR SD

moderate 20 20

Anxiety high 20 20

extreme 20 20

 

Results of posttest anxiety scores may show no interaction (parallel lines) or interaction (non-parallel lines)

 

 

 

  1. provides the economy of a single design rather than separate designs for each IV and allows researcher to investigate interactions
  2. reduces error variance and gives more power to test

 

    1. adds complexity: must look at main effects and interactions
    2. if added variables are unrelated to DV – may decrease power of the test (however if they are unrelated, you can remove them from the model)

 

Dependent samples designs:

 

 

Repeated measures designs (referred to as within measure designs):

 

S1 X1O – X20 … XKO

S2 X1O – X20 … XKO

. . . .

. . . .

. . . .

Sn X1O – X20 … XKO

 

where you have K experimental treatments and n subjects

 

sex

treatment

M1

M2

M3

M4

female

SD

10 Ss

randomly

assigned

measured 4X

 

PR

10 Ss

randomly

assigned

measured 4X

male

SD

10 Ss

randomly

assigned

measured 4X

 

PR

10 Ss

randomly

assigned

measured 4X

 

 

    1. can make the test more powerful (another source of variance is included in the model

 

 

    1. lots of data (more complicated)

 

Counterbalanced Designs:

 

 

RG1 O1 X1 O2 X2 O3

RG2 O4 X2 O5 X1 O6

 

1 2 3 1 2 3 4

3 1 2 4 1 2 3

2 3 1 3 4 1 2

2 3 4 1

 

 

    1. equal number of rows and columns
    2. each number is found only once in each row and once in each column

 

 

Sex

Subject

Time

 

 

T1

T2

T3

MALE

S1

X1

X2

X3

MALE

S2

X3

X1

X2

MALE

S3

X2

X3

X1

MALE

S4

X1

X2

X3

MALE

S5

X3

X1

X2

MALE

S6

X2

X3

X1

FEMALE

S7

X1

X2

X3

FEMALE

S8

X3

X1

X2

FEMALE

S9

X2

X3

X1

FEMALE

S10

X1

X2

X3

FEMALE

S11

X3

X1

X2

FEMALE

S12

X2

X3

X1

 

 

    1. more power
    2. high levels of experimental control (increased internal validity)
    3. can use fewer subjects

 

 

    1. very complex, especially as the number of IVs increases
    2. time consuming
    3. ceiling and floor effects – since within subject designs measure people several times, their scores may be maximized or minimized prior to the completion of the experiment

 

Designs Extended in Time:

 

 

RG1 X1 O1 – O2 – O3

RG2 – O4 – O5 – O6

 

Interpretation Practice

 

RG1 O1 X1 O2

RG2 O3 X2 O4

RG3 O5 X3 O6

RG4 O7 -- O8

 

  1. Results: O1 O2, O3 O4, O5 O6, O2 = O4, but O4 O6 and O1 = O3 = O5 = O7 = 08
  2. O1 = O3 = O4 = O5 = O6 = O7 = O8, but O1

O2

 

 

RG1 X1 O1 O2

RG2 X2 O3 O4

RG3 – O5 O6

 

  1. O1 = O3, but O3 and O1 O5, and O2 = O4 = O6
  2. O1O3, and O1 and O3O5, and O2O4, and O2 and O4

O6, but O1 = O2, and O3 = O4, and O5 = O6

 

RG1 O1 X O2 O3

RG2 O4 X – O5

RG3 – X O6 O7

RG4 – X – O8

RG5 O9 – O10 O11

RG6 – – – O3

 

  1. O3 O5, and O3 and O5

O11

 

Quasi-Experimental Designs

 

 

    1. lack of random assignment possibly introduces problems with the validity of an experiment (internal and external)
    2. internal – differential selection of subjects
    3. external – selection bias

 

 

 

X1 O1

 

lacks validity. Why? (do not know if treatment had an effect)

 

- posttest only nonequivalent design

 

G1 X1 O1

G2 X2 O2

G3 X3 O3

 

 

 

 

O1 X O2

 

 

better designs

 

 

G1 O1 X1 O2

G2 O3 X2 O4

G3 O5 O6

 

 

 

G1 O1 O2 X1 O3

G2 O4 O5 X2 O6

G3 O7 O8 O9

 

You can also enhance by giving treatments that are expected to cause changes in different ways

 

G1 O1 X+ O2

G2 O3 X- O4

 

 

 

Cohort treatments

 

 

 

 

O1 __ __ __

X O2

 

 

O1 O2 __ __ __

O3 X O4

 

Time series:

 

 

G O1 – O2 – O3 – O4 – X – O5 – O6

 

possible patterns

 

** draw examples (wiersma 143) **

 

 

G1 O1 – O2 – O3 – O4 – X – O5 – O6

G2 O7 – O8 – O9 – O10 – – O11 – O12

 

 

 

G O1 – O2 – X – O3 – O4 – X – O5 – O6 – X – O7 – O8

 

 

 

    1. useful where there are periodic fluctuations
    2. useful when there are naturally occurring situations for multiple testing

 

 

- complex analysis

 

  

Survey & Ex Post Facto Designs

Survey

 

Longitudinal design: involves data collection over time (i.e., data collection at two or more time periods) and at specified points in time (can be short periods or long periods)

 

Cross-sectional designs: involve collection of data at one point in time from a random sample representing some given population at that time or from more than one sample representing two or more populations (Wiersma)

 

Design

Population

Sampling

Longitudinal

 

 

Trend

General

Random samples at each data-collection time

Cohort

Specific

Random samples at each data-collection time

Panel

General or Specific

Same random sample used throughout

 

 

 

Cross-sectional

General or Specific & may include

Random samples from all populations at one timepoint

 

subpopulations*

 

* if 2 or more subpopulations studied simultaneously -- parallel-samples design

Methodology of Survey Research

 

 

Steps in Conducting a Survey

 

Definition of the research problem

1. Planning

Operational definition of variables

 

Literature review

 

Development of survey design

 

Definition of population

2. Development & Application

Identification of subpopulations

of Sampling Plan

Detailed sampling procedures

 

Select the sample

 

Develop items or select instrument

3. Construction of Interview

Development of anticipated analysis procedures

Schedule or Questionnaire

Pilot run

 

Revisions of items

 

Training of interviewers, observers, or testers

4. Data Collection

Conduct interviews, administer questionnaires or tests

 

Follow-ups

 

Initial tabulation and coding

5. Translation of Data

Coding

 

Technical preparations for analysis

6. Analysis

Separate analysis of individual items or groups of items

 

Synthesis, results interpreted

7. Reporting Conclusion

 

 

 

Questionnaires

  1. Have items relate directly to research problem, question, or hypothesis
  2. Items should be clear and unambiguous. Avoid jargon or vague terminology
  3. Include only one concept per item (use Michele’s survey for example)
  4. Avoid leading questions
  5. Avoid questions loaded with social or professional desirability
  6. Avoid (when possible) questions that demand delicate or personal information
  7. Request only information that the respondent can provide. Items should fit the informational background of the respondents
  8. Make reading level appropriate
  9. Shorter items better than long items - simple better than complex
  10. When requesting quantitative information, ask for specific number rather than an average (e.g., "how many times did you make yourself vomit in the last 2 weeks" rather than "on average, how many times per month do you make yourself vomit")
  11. Response options should be mutually exclusive and exhaustive
  12. Avoid negative items and never use double negative items (e.g., "which of the following symptoms do you not have")

Item format

a. forced choice or selected-response

b. open-ended

a. being regarded positively by others

b. expressing appreciation

c. being consulted on an issue of importance to the respondent

interview surveys

 

N=1

    1. nonbehavioral designs (case study and intensive ssd)
    2. behavioral designs

Designs


OOB

 

 

 

 

 

 

 

 

 

A B A

 

Baseline (A)

 

 

Treatment (B)

 

 

 

 

 

 

 

O1

 

O2

Ok+1

 

O2k

 

 

 

 

 

 

 

 

 

 

 

 

 

TA

 

 

TB

 

Ta = Tb

 

Baseline (A)

 

 

Treatment (B)

 

 

Baseline (A)

 

 

 

 

 

 

 

 

 

 

O1

 

O2

Ok+1

 

O2k

O2k+1

 

O3k

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

TA

 

 

TB

 

 

TA

 

ABAB

Randomized AB designs

Multiple-baseline designs

Multiple-baseline across subjects

Multiple-baseline across situations

 

Case studies

 

Qualitative designs

Ethnographic research

** note on syllabus, if ethnographic research, the first step of the qualitative proposal would be a problem statement **

  1. Is research understandable and reasonable?
  2. Do participants benefit?
  3. Triangulation
  1. "importance of findings"
  1. identification of the phenomenon to be studied

b) identification of subjects

c) hypothesis generation

d) data collection

    1. active participant: observer assumes the role of a participant
    2. privileged observer: observer does not assume the role of a participant but has access to the relevant activity for the study
    3. limited observers: used when opportunities for observation are restricted and other data collection techniques take precedence

 

triangulation involving multiple data sources

counselors faculty

 

students

 

 

triangulation involving multiple data collection procedures

student observations interview counselors

 

look at student records

 

Analysis

Drawing conclusions

 

 

Grounded theory

1. data collection

2. categorization

 

3. memoing

4. parsimony

5. writing the theory

 

Analogue Research

  1. Naturalistic vs. experimental approaches to research
  2. Counseling analogue is an experimental simulation of some aspect of the counseling process involving some manipulation of the counselor, client, or process -- miniature therapy or simplification strategy
  3. Experimental control - High: Generalizability - Low
  4. Analogues fall on a continuum from high to low resemblance to the counseling situation

Process Research

  1. Process Research attempts to characterize what changes occur during counseling
  2. Process research can involve intensive single-subject, within subjects, and between subjects designs
  3. Process research attempts to:
    1. Describe changes in the client, counselor, group, family, or interaction over time.
    2. Specify changes in the behavior or actions of those listed above over time.
    3. link one or more of these process variables to outcomes.
  1. What to measure
    1. content of session: topic or subject
    2. what speaking is done: by whom? words used.
    3. how speaking is done: non-verbals
    4. counselors’ intentions. why did you do that?
    5. reactions? what happened when counselor/client said/did that?
    6. quality? how helpful? good session?
    7. relationship?

5. The book has an excellent reference section on different instruments that can be used for process research.

 

Outcome Research

    1. no-treatment control group (does not consider the placebo effect)
    2. placebo control groups (often, placebo groups do not provide the "expectancies" of the treatment group)
    3. alternate therapy control groups (meta-analysis has found that often results of studies using alternate therapy are tied to the researchers expectations)

Defining Variables and Collecting Data:

 

  1. conditions or levels of the IV: need to determine the distinct levels or categories of the variable
  1. adequately reflecting the constructs designated as the cause in the research question -- is the IV adequately defined and does it accurately represent what you want it to

  1. limiting the differences between conditions -- conditions should differ only on the dimension of interest, if more differences are present the variable is confounded

a. make a logical argument that the confound is unlikely to have an effect

b. limit the generalizability of the study (e.g., in our counseling example we may assign new counselors to a condition and provide them with equal training, but our results would only generalize to new counselors)

4. establishing the salience of differences in conditions -- differences between conditions of the IV must be noticeable, but not too noticeable

Interpreting Results (relating to IVs)

Dependent Variables

Multiple Dependent Variables

Reactivity

 

 

Methods of Data Collection

 

  1. self-reports – the subject reports on his or her behavior or thoughts
  1. ratings of other persons and events – someone rates characteristics of a person or an event

 

  1. behavioral observations – trained raters record behaviors

 

  1. physiological indexes – measuring biological responses to infer psychological states

-

  1. interviews
  1. projective techniques – inferring personality traits from ambiguous stimuli
  1. unobtrusive measures – observing subjects without their awareness that they are being measured

 

 

Sampling

  1. define a target population
  2. create a sample pool
  3. select subjects
  4. establishing validity in the absence of random sampling
  5. determining the number of subjects to use
  1. define a target population

2. create a sample pool

  1. select subjects
  1. establishing validity in the absence of random sampling

 

5. determining the number of subjects to use

Using factorial designs to increase external validity

 

**** draw example of factorial design (2X2) – make a box representation and plot an interaction that shows how an interaction can be significant when a main effects (tx) is not ****

***** use example: frequency of misbehaviors in classroom

 

treatment A

treatment B

males

16

2

females

2

16

Xbar = 9, 9 treatment effect and interaction

 

treatment A

treatment B

males

16

5

females

15

4

treatment effect and gender effect

Specific sampling designs

Stratified Random Sampling

n/N = n1/N1 = n2/N2 = … + nk/Nk

n1 + n2 + … nk = n and N1 + N2 + … Nk = N keeping the proportions the same

Bias

Sources of bias

1. experimenter & investigator

2. subject bias