Ways of Knowing

• Tenacity: whatever belief one firmly adheres to is truth: something is true because it has always been true.
• Authority: if someone with authority says it's true, it must be true
• A priori: intuition - common sense says it's true
• Scientific method: empirical testing
• Personal experience

- Above by Charles Pierce - a 19^th century mathematician

- Discuss these ways of knowing – what are the strengths and weaknesses of them.

Ways to Make Informed Decisions

• Rely on personal experiences
• Consult an expert authority
• Review the literature
• Conduct research

The Scientific Method

1. identification of problem

2. form a hypothesis

3. deductive reasoning- decide on procedure

4. data collection and analysis

5. derive conclusion

Note: We never prove a hypothesis -- confirm or fail to confirm

Reliability

• in order to learn something from a set of scores or measures, the scores must vary (e.g., if you want to see the effect of computer guidance on students responsiveness to counseling, the measure of student responsiveness must be different for students in the group)
• Whenever scores vary across conditions (e.g., computer group vs. non-computer group) some of the variance is hoped to be due to the condition
• however, some of the variance is always due to error (imprecise measurement or unexplained factors that influence scores)
• in the language of classical test theory, observed scores are made up of the "true" score plus error:

where X = an observed score

T = a true score free from any error

e = error

• error may be due to systematic or random factors
• systematic – confounds
• random – non-systematic errors that can occur by chance or poorly designed measures
• non-systematic means that it while the scores are not precise measurements, they do not cause scores to systematically be higher or lower than the collective true scores
• The reliability of a set of scores is the degree to which the scores are due to systematic rather than chance factors – although it does not tell us what the systematic variance is due to
• Reliability also is defined as the consistency or reproducibility of a set of scores – how well can a test measure the same scores on different occasions
• going back to the CTT model we can also describe the model as:

the observed variability of a set of scores is equal to the true variability plus error variance (random error)

r_xx’ is an estimate of the reliability of a set of scores

- there are methods of estimating reliability that are not bound to the test-takers but these methods are not frequently used (IRT methods)

• Ways of estimating reliability

1. internal consistency
2. stability
3. alternate forms
4. interrater reliability

• Internal consistency reliability – measures of the homogeneity of items – people who measure high on some construct will tend to answer items in the same direction (if items are coded in the same direction) – people who measure low will answer in the opposite direction
• e.g., you have a scale to measure depression – items are coded 1-5 (strongly disagree to strongly agree) – high numbered items are indicative of depression – depressed people will have mostly higher number responses and non-depressed people will have low number responses
• these methods of estimating reliability require only one test administration to a group of people
• common methods:

1. Cronbach's alpha
2. Kuder Richardson 20 (KR20)
3. Kuder Richardson 21 (KR21)
4. Split-halves – correlate to halves of a test; there should be a positive correlation

• Stability – measures the correlation between tests given at different time points (test-retest)
• used only if you expect the construct being measured to remain stable over time (state vs trait anxiety)
• if this measure is used, you need to also report the time period
• Alternate forms (Parallel forms) – using two similar tests
• requires that the two tests measure the same construct
• do not see this in the literature very much
• Interrater reliability – required when a measure is not scored objectively, but subjectively
• e.g., Bender-Gestalt used to diagnose depression – have several experts make diagnosis and correlate the results – high correlations indicate good reliability
• requires that raters use standardized protocols and often means extensive training
• Reliability is essential in establishing validity – but reliability does not show validity – reliability can tell you how precisely you are measuring something, but not what you are measuring

**** do bullseye example *****

• Because reliability is tied to the group it was normed on, groups outside of the norm do may not be adequately measured by the test
• e.g., PF16 measures personality characteristics of normal people – people who are experiencing psychosis will not be accurately measured by the test (ceiling effect)
• the MMPI measures personality in non-normal subjects – it does not do well in measuring personality characteristics of normal populations (floor effect)
• Factors that effect reliability

1. test length – longer tests will be more reliable, but tests that are too long may cause test-takers to become fatigued or irritated – a good balance needs to be found
2. homogeneity – As with any correlation, heterogeneity among a group increases variance and the resulting correlation. In reliability estimates, as group homogeneity increases, true score variance decreases and the resulting proportion of true score to observed score variances decreases (Crocker & Algina, 1986).
3. item difficulty – related to homogeneity; if items are to difficult or too easy there will not be variance among the test-takers scores (e.g., everyone get a 100%)

Number of alternatives – the fewer the number of alternate answers – the better the reliability (e.g., t-f tests will tend to be more reliable than

Problem statement

Literature Review

Definitions

Specific research question

Significance of Research

Experimental Survey/Ex Post Facto Qualitative

Hypothesis Hypothesis Hypothesis

Subjects/Sampling Sampling Focusing

Treatment Instrumentation Sampling

Data Gathering Data Gathering Instrumentation

Data Analysis Schedule Iterations

Schedule Schedule

CW4B

USING APA STYLE --

THE MOST COMMON MISTAKES

1. Inventing your own rules for format and reference lists.

APA is a very precise style. Check the manual for the rules which are

comprehensive. Look at the samples provided. Ask questions if you run across

anything which does not fit APA rules.

2. Using incorrect margins.

APA uses 1 inch margins on all sides. The bottom margin may be adjusted if

needed, for example to avoid putting a heading on the last line of the page or

to avoid putting the last few words of a paragraph at the top of the next page.

Page headers go inside the top margin.

3. Using an author's first name or using gender specific pronouns.

APA requires nonsexist language and references at all times. Proof your work

carefully. There is no reason to ever identify the gender of a researcher or

author. 4. Formatting the title page incorrectly. Look at the samples and the

manual to be sure. The page header and the running head are two different

things.

5. Not formatting the reference page correctly. Follow the manual closely.

The reference list is in alphabetical order; check the manual if there is a

question. The reference list is not a bibliography; there should be a

correspondence between the reference list and the references in the text.

6. Using quotations.

Avoid direct quotes unless absolutely necessary. Direct quotes are rare in APA.

Summarize and paraphrase instead.

7. Using incorrect spacing.

Double space the text. All text is flush left. Do not hyphenate words. Check

spacing between headings and text.

8. Incorrect capitalization in a book or article title.

Caps are used only for the first word, proper nouns, and the first word after a

colon.

9. Putting too much information in the reference list.

Use an issue number only when warranted. Give month, season, or exact date only

when warranted. Follow the manual.

10. Using first person.

Never refer to yourself (I, we, me, etc.). You may have to resort to passive

voice. Check the manual for hints on how to strengthen your writing style.

11. Using the wrong verb tenses.

Double check verb tenses. In general, when referring to research conducted in

the past, use past tense ("Smith concluded.."). When referring to the present

state of knowledge or theory, use present tense ("These studies suggest...").

When talking about your proposed methods, use future tense ("The data will be

collected..."). Look at the samples.

• Review sources of ethics: ACA, APA, NASP, ASCA
• 5 fundamental ethical principles

1. nonmaleficence: "do no harm" -- in planning research you must be aware of harm or potential harm that may come to subjects

• may be the most basic and important principle
• if you must make the decision of helping others vs. potentially harming some, you are primarily obligated to do no harm (e.g., want to determine if you have developed a good therapy for use with severely depressed. To best test your method, you would need a control group -- however, denying treatment to a group would be potentially harming and you would be obligated to not use a no-treatment control group)

1. beneficence: "do good for others"

• to do good, we must be competent in what we practice (clinically or research)
• to do good also demands that we do research to improve our knowledge of effective practices
• to do the best for our clients, we should be aware of current research and be able to evaluate good and bad research

1. Autonomy

• individuals have the right to choose freely and in an informed manner whether to participate in research "informed consent"

1. Justice

• Justice implies fairness: we cannot discriminate unjustly because of sex, race, ...; however we can treat differently if such treatment is warranted by differences
• e.g., cannot deny treatment or provide different levels of treatment because of clients racial background, however, we may provide different treatments or methods of treatment if such treatments are more likely to be effective.
• these determinations are made by people with biases and prejudices -- must use caution in providing different types of treatments but still use best practices if different for different groups
• justice also applies to credit for work done (articles, presentations, ...)

1. Fidelity: faithfulness

• do not engage in deception (research has often relied on deception, this is a loaded principle for research) see g2b
• if you promise to provide results -- do so
• you are obliged to stick by agreements of research with clients

Issues in research ethics

As researchers we are expected to accurately report knowledge gained from research and to prevent the misuse of such knowledge

• The researcher has the responsibility of producing valid results.
• This requires knowledge of research principles
• The researcher is ultimately responsible to see that subjects receive ethical treatment and that reported results are reliable and valid.

• reporting accurate results is often challenging. who want’s to hear insignificant results – tendency to report partial results or stretch findings based on researchers bias.
• If you as the researcher do not follow procedures accurately and consistently you may produce invalid results
• You are obligated to provide limitations and problems with your design and analysis
• You have the responsibility to store your data for an amount of time so that others can check it out (see g3e)
• you cannot make up data or results without reporting that

• Contributors to research should get publication credit (see and discuss section G4)
• you must recognize and reference the works of others in your research. Direct and indirect plagiarism is prohibited

• Federal law has mandated that behavioral research done on humans must be reviewed by Institutional Review Boards (IRB)
• at OSU, research on humans must be reviewed and approved by the Human Subjects Committee

• It is the researcher responsibility to provide information that increases the knowledge in the field while preserving the dignity and safety of the subjects
• must identify potentials of harm ( e.g., is filling out a test going to create anxiety or depression?)
• How do you assess and deal with the potentials for harm?

1. estimate the cost/benefit of the study -- do the benefits outweigh the costs; are the potential benefits greater than the potential harms?; who benefits, the client or the society

• hard for the researcher to assess this on own

1. minimize the risks -- are there other ways of designing research that would reduce potential harm?

1. assess risks through pilot studies (with colleagues) or role-playing

1. select subjects that are less likely to be harmed (e.g., don’t select severely depressed clients for participation in a study involving criticism from others)

1. ALWAYS: consult with others

Informed consent:

• researchers are obligated to get informed consent from the participants
• informed consent requires that the subjects understand their obligations and the risks associated with the study before research begins
• client must have the capacity to give consent
• children must have parents consent and children must give assent (agreement)
• clients must not be pressured to participate and may terminate participation at their discretion.
• DOCUMENT clients consent

• although counselors and psychologists are to "avoid deception" there is not a clear guideline on when to avoid.
• some research requires deception (e.g., story telling to kids to see if their memories changed)
• if deception is to be used you must:

1. determine if the potentials for harm

2. determine if other designs could be used

3. CONSULT with others

Confidentiality

• confidentiality vs. anonymity
• must not give information in results that could break confidentiality
• must inform client on who will be privy to confidential information
• the usual limits to confidentiality

Treatment issues

• control groups, placebos, waiting list groups
• often not needed since there is some agreement on the benefits of therapy in general
• must be sure if clients have treatment delayed that the potential for harm is minimal
• sometimes use natural wait-list groups

• Survey research is one of the most widely used research methods in the social sciences
• It is done in a variety of forms from "status" surveys to determine the status quo to ex post facto studies that focus on the relationship of sociological and psychological variables as they occur in natural settings
• Ex post facto means "from a thing done afterwards" -- it is research done after the fact -- research takes place after groups or conditions have been formed

• The purpose of survey research is to describe, explain, or explore phenomena (Babbie, 1979)
• Surveys are frequently seen today in the form of political polls -- (ask students to give other examples of surveys)
• Counselors may use surveys to describe or explore attitudes, opinions, frequency of occurrences, ... within a specified population
• It is descriptive in nature and is generally used to explain the relationship between variables and some phenomena
• data can be collected through interviews, phone interviews, internet, mailed forms, questionnaires.
• Surveys are basically of 2 types: longitudinal and cross-sectional
• these types are distinguished by (a) point at which data collection takes place and (b) the nature of the sample
• surveys are done with samples -- when the entire population is used we have a census

Longitudinal design: involves data collection over time (i.e., data collection at two or more time periods) and at specified points in time (can be short periods or long periods)

• trend study: a longitudinal study in which a general population is studied over time.
• random samples are taken at different time points -- different samples, but all representative of the population (e.g., political polls that determine what party is likely to win an election -- done several times leading up to the election)
• cohort studies are also longitudinal and similar to trend studies
• however they are used to measure variables over time of a specific population
• e.g., if you want to measure NY counselor attitudes towards an ethical issue over 10 years -- randomly sample population every year and survey on attitudes -- however with each sampling, only use the counselors present at the time of the original sampling
• trend and cohort studies measure change in a group over time, but because random samples are selected at each time point, any individual changes are not measured -- one does not know who is causing the changes
• Panel studies collect information on a sample of individuals at different times
• the sample is called the panel -- which should be randomly selected at the onset of the study
• panel studies look can be used to establish an ordering of patterns that is helpful in establishing cause and effect
• for example, a panel of mental health consumers are question questioned on community mental health services -- if people in treatment a long time report good services, are they reporting that because the got helped, or was their positive attitude a determinant of their success. If we can measure the same people over time we can see how perceptions change and establish an order which helps in determining cause and effect
• a major disadvantage of panel designs is attrition
• it is also very time consuming for participants and researcher

Cross-sectional designs: involve collection of data at one point in time from a random sample representing some given population at that time or from more than one sample representing two or more populations (Wiersma)

• cross-sectional designs can’t measure individual change because subjects are measured only once
• c-s designs can look at the difference between defined groups between two populations -- parallel-samples design
• parallel-samples designs usually occur in c-s surveys, but can be done in longitudinal studies
• e.g., (p-s) sample three groups of mental health clients -- those that go to counselors, psychologists, and social workers -- give them the same questionnaire and compare results.

* if 2 or more subpopulations studied simultaneously -- parallel-samples design

Methodology of Survey Research

• first step is to define research problem and begin developing survey design
• variables to be included in the research design must be operationally defined, and the investigator should have information about the relationships between the variables -- this information is needed for constructing the survey items
• next step -- develop sampling plan (unless you will survey entire population)
• define population
• units identified for sample selection
• the sample must be selected in such a way as to make valid inferences to the population and any subpopulations
• sampling procedures can be very complex and require much effort and resources
• preparation for data collection

• if doing interviews or questionnaires -- must construct instruments
• if using tests or inventories, it is likely there are instruments available -- may have to train observers or testers
• identify the types of data you will collect and how you will analyze it
• initial drafts of questionnaires should be tried out (pilot or trial run) on a small number of people. See problems with questionnaire.
• Collect data
• stick to sampling plan
• if using interviews -- periodically have two interviewers do the same person -- get measure of agreement (interrater reliability)
• also have one interviewer redo interview (by tape) and check internal consistency (intrarater reliability)
• analyze
• may be quantitative (statistical) or qualitative (descriptive)

Steps in Conducting a Survey

• possible problems with surveys
• survey may be poorly designed
• nonresponse bias
• failure to provide follow-up
• failure to provide synthesis of entire analysis

Questionnaires

• much effort is required develop good items and getting people to respond
• item construction -- general guidelines

1. Have items relate directly to research problem, question, or hypothesis
2. Items should be clear and unambiguous. Avoid jargon or vague terminology
3. Include only one concept per item (use Michele’s survey for example)
4. Avoid leading questions
5. Avoid questions loaded with social or professional desirability
6. Avoid (when possible) questions that demand delicate or personal information
7. Request only information that the respondent can provide. Items should fit the informational background of the respondents
8. Make reading level appropriate
9. Shorter items better than long items - simple better than complex
10. When requesting quantitative information, ask for specific number rather than an average (e.g., "how many times did you make yourself vomit in the last 2 weeks" rather than "on average, how many times per month do you make yourself vomit")
11. Response options should be mutually exclusive and exhaustive
12. Avoid negative items and never use double negative items (e.g., "which of the following symptoms do you not have")

Item format

• types of items for questionnaires (discuss advantages and disadvantages [time, likelihood of response, interpretation, freedom of response])

• pilot run
• allows you to uncover deficiencies with items
• does not need to be random sample
• can be done on group that is familiar with survey topic
• suggestions can be made
• items that are unclear can be revised
• items that provide little information can be revised (e.g., having people respond to the number of years in the field [0-5, 5-10, 10+] -- if most people were in the 0-5 group, you could revise the responses to [0-2, 3-5, 6-8, 9-11, 11+])
• cover letter
• should be straight forward explaining the purpose and the potential value
• assure reader that group results are to be used and people will not be singled out
• confidentiality assured (may be anonymous - but remember it is difficult to follow-up with anonymous respondents)
• use professional letterhead
• if possible, have someone of importance sign letter
• deadline for responses
• questionnaire format
• should be attractive and easy to read
• don’t make it too long
• place questions in an order that will keep respondents attention
• give clear and concise instructions
• give examples if instructions are complex
• the cover letter should set the stage for responding - initially begin survey with questions directly related to research problem
• put demographics near end
• put open-ended questions at the end of survey
• give respondent an opportunity to write open-ended responses at end even if you do not plan to use that information
• number pages and items
• give name and address of person to respond to and provide sas envelope
• increasing response rate
• generally consider 70% a minimum with professional groups
• less is tolerated with general populations
• people are more likely to respond if the time and effort is low in comparison to perceived rewards -- to increase perceived rewards, the researcher can offer these rewards (Dillman, 1978)

• monetary responses do work but they are costly
• Hopkins and Gullickson (1989) found monetary rewards equally effective for professional and non-professional populations and that enclosed gratuities were more effective than rewards after completion of survey
• attractiveness and professional look will increase response
• contacting respondents prior to mailing of questionnaire
• realistic time ques (e.g., 30 minutes to fill out)
• follow-ups necessary
• timed to arrive at the respondents’ addresses a few days after the deadline for return specified in the cover letter
• should be planned for in advance and more than one may be necessary
• letter should be pleasant but firm
• Jackson and Schuyler (1984) found businesslike follow ups more effective than "cute" reminders
• if possible, include new questionnaire in follow-up
• determining sources of non-response
• important to know sources because it is important to identify possible bias in collected responses
• look at demographic data - may help you determine characteristics of those who did not respond
• if non-respondents are identified as certain groups, attitudes associated with those groups may be missing - leading to biased results

interview surveys

• more time consuming
• questions again may be open ended or select-response
• questions should be the same for all respondents
• interviewers must be trained and procedures standardized

• Heppner et al. use the term passive design to describe a type of ex-post facto research (measure subjects on some characteristics and correlate) -- this comes under the definition of survey research given here
• Classification research -- use cluster or factor analysis to find out underlying constructs -- useful in exploring or confirming theories or for finding latent constructs (characteristics we can not easily observe)

N=1

• Single subject designs are often difficult to interpret because of the many sources of bias
• However, ssds are often more practical and can be designed to give us valuable information
• as counselors, you will often have the opportunity to use ssds because of the number of clients seen individually
• subjects are not usually selected on a random basis, so traditional behavioral models of ssds are considered quasi-experimental
• History in psychology:
• early scientific in the human sciences was done with single subjects (e.g., Freud, Skinner)
• many of the early (early 20th century) psychological studies based on case studies were very flawed - conclusions were based on wholly uncontrolled studies of single persons (e.g., Freud’s theories based on case studies -- Little Hans ...
• as the use of statistical knowledge spread to the psychology field in the mid 20th century -- the sophistication of ssds increased
• most current research is based on group comparison designs
• however the group comparison model often fails to recognize individual differences or unique subjects
• Heppner classifies ssds into two groups

1. nonbehavioral designs (case study and intensive ssd)
2. behavioral designs

• we will begin by looking at traditionally behaviorally oriented designs (intra-subject designs)
• intra-subject designs examine the relationship between 2 or more variables within 1 or a few subjects
• ssd commonly involve repeated measurements and use the single-variable rule -- changing only one variable (the treatment) at a time
• this means that only 1 variable, the treatment is changed during the period in which the experimental treatment is applied
• during the traditional or baseline treatment and the experimental treatment, all other conditions (e.g., length of time or number of measurements) are kept the same
• baseline: the period of time in which the traditional treatment or normal condition is in effect -- must be stable in order to make inferences about subsequent treatments
• target behaviors: the dependent variable of the design - the behavior that is being measured to determine if change occurs
• generally, measurements are made over a period of time and different phases are compared to each other (i.e., baseline - treatment or treatment 1 - treatment 2)

Designs

• the letters A & B are used to represent conditions; A indicates the baseline condition and B indicates the experimental treatment condition -- because there are no groups used, there is no group notation
• AB design:
• subject is observed under baseline condition until the dv stabilizes
• experimental treatment is introduced and subject observed the same number of times
• interpretation based on the assumption that the observations would not have changed without the introduction of the treatment
• many threats to validity: history and maturation are main concerns to internal validity
• draw example on board (general case and example [out of seat behaviors vs relaxation treatment])

OOB

A B A

Ta = Tb

• ABA
• the ABA design is an extension of the AB design
• it adds another baseline condition (also called reversal or withdrawal design)
• duration and number of observations kept the same for all 3 conditions
• enhances the internal validity because the pattern of results is extended
• give example on the board (disruptive behaviors vs. relaxation)

ABAB

• treatment is administered - withdrawn - and readministered
• strengthens causal argument if treatment appears to cause an effect that is reduced when it is withdrawn and enhanced when it is administered
• if trend does not return to baseline levels - then it is highly likely that other factors were influencing change (time or maturation)
• however, some treatments should not be expected to return to baseline (e.g., do grief counseling -- if you treat the client and stop treatment, we would not expect the level of grief to return to baseline if the treatment was effective)
• Heppner states that we can have no confidence in results if they do not return to baseline -- however, while many measures are not likely to return to baseline, we can also look at trends (draw examples)
• ** give example of relaxation introduced twice **

• Treatments phases are randomly placed in series of baseline phases
• AAABAABAAAABABABAAB
• helps detect if there is a "crossover" effect - if treatments effect subsequent baselines or if baselines effect subsequent treatments
• Heppner gives example of how to analyze with factorial ANOVA -- you do not need to know this
• Typical statistical analysis is to plot the data and observe trends
• especially when trends are not extremely evident -- the likelihood of the analyzer introducing bias is great
• statistical methods include repeated measures or time series analysis

Multiple-baseline designs

• modification of the ssds in which more than one subject, situation, or behavior is measured (or a combination of these)
• helps to establish causality (i.e., treatment is causing the changes)
• multiple baseline across behaviors (2 or more dvs)
• collect data on dvs and establish stable baseline
• apply treatment to 1 of the dvs: should have an effect on targeted dv but not the other(s)
• after treatment phase (T1 = T2) another treatment phase begins targeted at next dv
• e.g. positive reinforcement has an effect on problem behaviors
• dvs= out of seat, talking at inappropriate times
• 2 week baseline (measurement of frequency of out of seat and talking behaviors)
• 2 weeks (+) reinforcement for in seat behaviors (ignore talking)
• 2 weeks (+) reinforcement for not talking
• may designate a dv as control (never apply treatment to that dv)

Multiple-baseline across subjects

• two or more subjects treated separately -- important that subjects are independent so that treatment of one does not affect treatment of other
• e.g., relaxation exercise
• dv = misbehaviors in the class
• 2 subjects
• 2 week baseline
• 2 week treatment to S1
• 2 week treatment to S2

Multiple-baseline across situations

• same as across behaviors except that situations are changed
• .g. relaxation exercise has an effect on behavior
• dvs= problem behavior in school, problem behavior at home
• 2 week baseline (measurements at home and school)
• 2 weeks relaxation sessions in school (not home)
• 2 weeks relaxation sessions at home
• may designate a dv as control (never apply treatment to that dv)
• MB designs can easily become very complicated
• the above designs were using the AB logic although more complicated designs could be used (e.g., ABAB)
• MB designs tend to improve internal validity **why? better case for causality**
• all ssds have the benefit of being practical ways of testing new ideas and studying unique cases
• disadvantages include: extensive threats to internal validity, external validity not supported by design (must be argued logically)

Case studies

• frequently done in counseling and psychological research
• descriptive in nature
• typically, a selected case is examined and the course of a treatment (especially outcomes) is described
• case studies can become more quasi experimental depending on the amount of systematic, objective, and repeated measurements of the subject
• case studies may be helpful in
• developing a preliminary hypothesis for further explanation
• capitalizing on natural contexts
• providing in depth descriptions
• focusing on uniqueness of individuals
• disadvantages
• lacks any generalizability
• researcher bias
• example "Dibs"

Qualitative designs

• may include case studies
• historical research is typically qualitative

Ethnographic research

** note on syllabus, if ethnographic research, the first step of the qualitative proposal would be a problem statement **

• roots in anthropology -- over last couple of decades, qualitative research has become increasingly popular in psychology, counseling, and educational research
• ethnography is defined in the dictionary as "a branch of anthropology dealing with the scientific description of individual cultures"
• ethnography is on the qualitative end of the qual-quant continuum -- but this does not mean that quantitative analysis must be excluded from ethnographic research
• ethnographic research involves field research and requires contextualization -- or the interpretation of results in the context of the data collection -- because of this it lacks generalizability to people "outside" of the context.
• while the other types of research that we have looked at thus far require some kind of hypothesis prior to conducting the research -- ethnography assumes that hypotheses will emerge as the data collection occurs
• data collection is inductive vs. deductive
• the researcher attempts to set aside any preconceived notions that might bias the data interpretation
• hypotheses may emerge and be refined or replaced as data collection progresses
• ethnographic research often does not start from a strong theoretical base and is not concerned with theory testing -- theory development is based on the data -- from this comes the name grounded theory
• the researcher does not stand apart from the research - rather he or she may become intimately involved with the subjects
• data collection generally involves ethnographic observation, oral histories (monologue), qualitative interviews (open ended and interactive), archival research, or critical incident reports (subject reports of significant events) (Hoshmand, 1989)
• the steps involved in qual research do not follow strict guidelines such as in quant research. the steps are usually integrated -- several steps may be done at the same time
• Validity

1. Is research understandable and reasonable?
2. Do participants benefit?
3. Triangulation

• Reliability

1. "importance of findings"

• typical steps in qualitative research - 2 methods (different than book)

1. identification of the phenomenon to be studied

• a statement of the phenomenon to be studied (e.g., peer interactions in racially mixed classrooms of a rural high school
• statement provides a starting point and usually implies a foreshadowed problem
• foreshadowed problems may include things such as:
• interaction among students across races
• interaction among students across sexes
• role of faculty in student’s social interactions
• these problems give the researcher a starting point but should not be considered restrictive - they can change as study progresses

b) identification of subjects

• usually subjects that are convenient -- any generalizability would be argued on a logical basis
• conditions and settings must be considered -- how many and which students will be observed? how long will they be observed? how often will they be observed?
• usually done on the basis of convenience, time limitations, ...

c) hypothesis generation

• is a continuing activity throughout an ethnographic study
• may be refined or changed based on data

d) data collection

• observation: the observer may be identified as a researcher or may be in a disguised role
• Wolcott (1988) distinguishes between 3 participant-observer styles

1. active participant: observer assumes the role of a participant
2. privileged observer: observer does not assume the role of a participant but has access to the relevant activity for the study
3. limited observers: used when opportunities for observation are restricted and other data collection techniques take precedence

• all data collection need not be with the participant observer role
• other sources of information including interviews, tapes, and surveys may be used
• sources of info should be primary sources
• whatever the role - observers try to be unobtrusive and not interfere with the normal activities (star trek’s prime directive)
• understanding and interpreting events requires the observer to attempt to experience the thoughts, feelings, and actions of the individuals under study
• triangulation: part of data collection that cuts across 2 or more techniques or sources -- a qualitative cross-validation
• basically, it is a comparison of info to determine whether or not there is corroboration
• do different sources of info or different data collection procedures converge?

counselors faculty

students

student observations interview counselors

look at student records

Analysis

• synthesizing information from all data sources
• statistical procedures not usually used
• heavily descriptive

Drawing conclusions

• tentative hypotheses, theories, and explanations are generated during the field-work, but researchers should not draw final conclusions prematurely
• lead to a successive approximation procedure of coming to conclusions

• systematically observing a phenomena as a way of generating a theory
• five steps

1. data collection

• subjects chosen to exemplify the phenomena
• may pick a comparison group to extend generalizability

2. categorization

• process of categorizing data
• start broad and narrow down to themes
• saturation is when new data conforms to existing themes

3. memoing

• record the process to aid in constructing categories and building theory

4. parsimony

• consolidate categories with ultimate goal of 1 "core" category

5. writing the theory

• should be believable
• comprehensive
• theory tied to data
• should be applicable

1. Naturalistic vs. experimental approaches to research
2. Counseling analogue is an experimental simulation of some aspect of the counseling process involving some manipulation of the counselor, client, or process -- miniature therapy or simplification strategy
3. Experimental control - High: Generalizability - Low
4. Analogues fall on a continuum from high to low resemblance to the counseling situation

• Advantages:
• Control over extraneous variables
• Allows the researcher to identify, quantify and manipulate specific levels of an investigated variable (e.g., counselor disclosure)
• Disadvantages:
• generalizability: high internal validity often leads to artificiality

1. Process Research attempts to characterize what changes occur during counseling
2. Process research can involve intensive single-subject, within subjects, and between subjects designs
3. Process research attempts to:

1. Describe changes in the client, counselor, group, family, or interaction over time.
2. Specify changes in the behavior or actions of those listed above over time.
3. link one or more of these process variables to outcomes.

4. What to measure

1. content of session: topic or subject
2. what speaking is done: by whom? words used.
3. how speaking is done: non-verbals
4. counselors’ intentions. why did you do that?
5. reactions? what happened when counselor/client said/did that?
6. quality? how helpful? good session?
7. relationship?

• Outcome research looks at the efficacy of counseling by comparing a treatment group to a control group or to different treatment types
• Types of treatment groups:

1. no-treatment control group (does not consider the placebo effect)
2. placebo control groups (often, placebo groups do not provide the "expectancies" of the treatment group)
3. alternate therapy control groups (meta-analysis has found that often results of studies using alternate therapy are tied to the researchers expectations)

• Measuring change: statistical vs. practical significance

• One of the main focuses of research is to establish a causal relationship between the IVs and DVs
• the selection and designing of the IVs is a crucial task in establishing these relationships
• Heppner outlines 4 concerns regarding the operationalizing Ivs

1. conditions or levels of the IV: need to determine the distinct levels or categories of the variable

2. adequately reflecting the constructs designated as the cause in the research question -- is the IV adequately defined and does it accurately represent what you want it to

• e.g., if you are doing a study comparing directive to non-directive therapy, do the therapy protocols accurately represent these modes

3. limiting the differences between conditions -- conditions should differ only on the dimension of interest, if more differences are present the variable is confounded

• e.g., if you are doing a study comparing directive to non-directive therapy and use 2 counselors (1 for each condition) -- therapy is confounded with therapist
• sometimes it is very difficult to get rid of these confounds so you may have to

• you want to make sure that different condition are similar in all ways (e.g., time, setting,...) except for the difference that you are investigating

• the differences between conditions must be great enough that any real differences between conditions can be found. e.g., if you design a study to investigate the how stress in a counseling situation affects a clients motivation to change and you have two conditions -- normal stress and increased stress. To increase the stress in one group, you tell them in the middle of a session that "you need to work harder if you want to get better". Is this stress situation significantly different than the normal stress situation to detect differences if they really exist?
• however, if the difference is too great, you risk having the subjects guess the research hypothesis – for example, if you are doing a study to rate client comfort with a counselor based on attractiveness and you send in one counselor (an attractive, well dressed counselor) and rate comfort than send in another counselor who has dirty clothes, unkempt hair, slouches and has a dirty greasy face, the subjects may guess the purpose and not respond naturally
• you can check the salience of your IVs with manipulation checks
• have the subjects tell you if they noticed differences (although you might want to use some subjects for this purpose alone)
• have independent raters – experts or naïve persons
• independent raters can also be used to check if other confounding factors are being measured – e.g., book example of picture of Hispanic and Anglo counselors; raters could determine if counselors equally attractive so that attractiveness does not confound ethnicity
• when using people to administer treatment, make sure adequate protocols are in place to insure treatments are the same

Interpreting Results (relating to IVs)

• If you have performed manipulation checks and found that the IV did not discriminate well, any statistically significant conclusion would be difficult to explain
• there is always the possibility of confounding variables, you just try to account for them and minimize their effect
• statistically non-significant results may be due to no actual differences between the treatment groups, inadequate statistical power, insensitive instruments, poorly done statistical tests, careless procedures, bias, and many other things
• Status variables: Variables that can’t be categorized as IVs (because they can’t be manipulated) but are still used to see effects on DV – e.g., sex, personality, race,…
• Can’t be used to show causation – just association

• Choosing and designing the DVs is critical
• if the dv does not accurately measure the construct you have defined as the effect, the study lacks both internal and external validity
• the first step in defining the dv is to have an adequately defined, concise research question and hypothesis
• the question should clearly state what is going to be measured as the effect(s)

• it is often desirable to use multiple dvs to measure a construct because tests usually do not represent the "whole construct" being measured
• also, experiments will often have and effect on more than one construct
• MDVs does require more complex statistical analysis

• You do not want the test to cause some kind of reaction in the test-taker – this can influence the results of the test in a systematic way
• e.g., sending a rater into a classroom – children will behave differently – you can overcome by desensitizing children to the rater before taking measurements

Methods of Data Collection

• Heppner et al. categorized data collection into 7 independent groups

1. self-reports – the subject reports on his or her behavior or thoughts

• advantages – easy to administer; simple for the test-taker to use; can access areas that would otherwise be difficult to measure; can have test-taker respond to hypothetical situations that may be impractical or unethical to test
• disadvantages – measurements are vulnerable to distortion by the test-taker (bias – e.g., may respond to look good, please the administrator; look bad, …); the test-taker may not be responding accurately to questions because they are unaware (e.g., subject may not report that they are depressed because they attribute their signs of depression to something else like lack of sleep)
• most often used measure in the field

2. ratings of other persons and events – someone rates characteristics of a person or an event

• ratings have many of the same benefits as self reports
• ratings also have the benefit of being based on some kind of expertise
• primary disadvantage is systematic bias – raters are reporting their perception of a person or an event (e.g., a rater rating counseling sessions may tend to see them all as poor while another may see them as all above average)
• systematic bias is also a problem when raters know what the test is about
• training, practice, multiple raters, and statistical techniques are used to reduce or account for the systematic biases of raters

1. behavioral observations – trained raters record behaviors

• much more objective than self reports

1. physiological indexes – measuring biological responses to infer psychological states

• can be more objective than self reports
• is not a direct measure of a construct
• expensive and machine dependent
• not used much in counseling research

-

1. interviews

• structured interviews vs. unstructured interviews (general themes)
• can get more depth to information
• in addition to self-reports, the interviewer can make impressions
• costly and time consuming
• unstructured interviews are difficult to standardize when doing quantitative research

1. projective techniques – inferring personality traits from ambiguous stimuli

• associated with psychodynamic approaches
• always a problem of bias – inferring personality characteristics from preconceived notions rather than measurement

1. unobtrusive measures – observing subjects without their awareness that they are being measured

• nonreactive
• often difficult to interpret
• sometimes unethical (getting info on a subject without their permission)

• meaningful research requires that subjects used for the experiment be representative of the population of which you are trying to make conclusions
• the generalizability of a study is based to a large extent on the sample you have selected
• Sampling Theory is concerned with how we select samples that represent our population of interest
• defining the population of interest is often difficult – populations are often changing and not fixed
• e.g., if you are doing an experiment on depression treatment; what group do you want to make inferences about? All depressed people who are likely to attend counseling? The number of depressed people is always changing and the reasons for depression are also changing.
• Sample – a subset of the population
• Bias – occurs when a sample systematically differs from the population
• e.g., if you if you are doing research on the effects of studying time on grades in college and only select seniors for the sample
• Bias – when all observations in a population do not have an equal chance of being selected
• Random selection insures against bias in a sample
• a random sample is a probability sample in that every population member has a nonzero probability of selection. In a simple random sample, this probability is the same for all population members
• e.g., a population has 100 members: if everyone has an equal chance of being selected the probability that a person is selected is 1/100 or 0.01.
• can use random number tables, draw number out of hat, or use random number generators
• It is usually impossible to have access to an entire population, so this type of sampling is impractical and rarely used
• instead: we
• define a target population
• create a subject pool
• select subjects
• establish validity in the absence of random sampling
• determine the # of subjects needed

1. define a target population
2. create a sample pool
3. select subjects
4. establishing validity in the absence of random sampling
5. determining the number of subjects to use

1. define a target population

• define the population you want to generalize to
• decide what characteristics you want to be represented

2. create a sample pool

• from the target population – determine who you have access to – often we use convenience sample

3. select subjects

• random selection is the best but often is not practical
• try to select subjects that can best represent the target population

4. establishing validity in the absence of random sampling

• external validity must be made on a logical argument basis
• randomly assign subjects to treatment groups – essential for internal validity

5. determining the number of subjects to use

• relies on power – will discuss later, but read this in book

• by using factorial designs, we can account for differences among people, settings, and designs
• including status variables makes the analysis more sensitive to differences in treatments

**** draw example of factorial design (2X2) – make a box representation and plot an interaction that shows how an interaction can be significant when a main effects (tx) is not ****

***** use example: frequency of misbehaviors in classroom

Xbar = 9, 9 treatment effect and interaction

treatment effect and gender effect

• select status variables based on theory – have an understanding of what may be related to the outcome – this comes from the literature and your understanding of the research problem

• we have already discussed simple random sampling (p=1/N)
• the sampling fraction is the ratio of sample size to population size – if you select a sample of 10 from a population of 100 the sampling fraction is 1/10 or 0.10

Stratified Random Sampling

• if a population is not homogenous, but is made up of many subpopulations, we can use stratified sampling to insure that we get subjects from each subpopulation
• equal allocation – if there are five stratas each sample from the population would be 1/5^th of the total sample
• proportional allocation – if strata size differs, we can hold the proportions constant across strata:

n/N = n1/N1 = n2/N2 = … + nk/Nk

n1 + n2 + … nk = n and N1 + N2 + … Nk = N keeping the proportions the same

• stratified sampling guards against wild samples and avoids overlapping among subpopulations
• cluster sampling: random selection of clusters from the larger population of clusters
• e.g., when you have natural clusters (city blocks) and you randomly select clusters – everyone in the cluster is used
• Sampling through an intermediate unit
• like cluster sampling except that after the clusters are selected – subjects from within the clusters are randomly selected
• 2 stage sampling
• the probability that someone is selected is then the product of 2 probabilities – the probability that a cluster is selected and the product of being selected within a cluster

Sources of bias

• investigator
• experimenter
• subject

1. experimenter & investigator

• Is it reasonable to assume that the investigator or experimenter can be totally objective? (free from bias)
• What should the investigator/experimenter do to guard against bias?
• experimenter attributes
• avoid using single experimenters for different levels of treatment
• if using more than one experimenter, check for any differences between experimenters
• describe your experimenters so that the study can be used as part of an overall database for future research and the experiment can be reproduced
• point out limitations
• experimenter/investigator expectancies
• try to keep experimenters blind to conditions (healing touch example and keeping surveyers blind to purpose of survey)
• assess how well experimenters have guessed the hypothesis of the study
• experimental procedures
• keep procedures consistent and well structured
• standardize procedures as best as possible (e.g., using counselor videos shown to subjects to insure same viewing experience)
• carefully train experimenters
• monitor experimenters throughout study

2. subject bias

• demand characteristics: cues in the research protocol that may make subjects respond in a certain way
• e.g., "the effects of alcohol abuse disorders on shame and guilt" -- title may make people respond in a certain way to alcohol abuse survey
• self-presentation: subjects may respond in some way to avoid being seen in a negative way
• motivation level: subjects may lack motivation to participate and only participate half-heartedly
• intellectual skills: not adequate intellectual skills such as reading
• psychological defenses: guarded or untrue responses based on real or perceived threat
• keep subjects blind to purpose if possible
• assure subjects that there are no right or wrong answers
• ensure confidentiality
• increase motivation with a reward (e.g., $, study results, an explanation of how the study will benefit the subject)
• use spot check items (e.g., inserting a question that tells subject to mark "strongly agree"
• evaluate reading level of subjects and tests