>>Welcome, everybody.
This is the March 2022 meeting of the CSR advisory
council.
I'm Bruce Reed.
I'm the deputy director at CSR.
I'll serve as the executive secretary, so I guess
that's the master of ceremonies and timekeeper is what
I'll be doing.
Before I go to introductions, I just want to remind
people of a few things.
So first of all, this is a public meeting.
The only people that you'll see on the screen are
yourself, the council members, plus the presenters,
Noni and myself and then the interpreters.
Since it's a small group, we would certainly like you
to keep your screens on.
I recognize that this goes from 10:00 in the morning
until 2:00 in the afternoon for some of you.
And we did not give you a decent break.
So if you've got to turn off the screen to, you know,
catch a snack, a short lunch, feel free to do so, but
then come back and join us visually.
The chat is shut off.
The Q&A is shut off.
So as I said the other day, just raise your physical
hand, raise your zoom hand or just speak up.
It's a small group.
So don't be shy about just interrupting during the
presentation if you'd like to -- if you have a question
or comment.
Okay.
So with that, let's go ahead and let's go through the
introductions. I will -- I'll just walk you through
this.
So I'll start off with a name.
If you could tell us, just introduce yourself, where
you're from, a couple words about your science.
So Dr. Cabassa.
>> DR. LEOPOLDO CABASSA: Hello, everyone. I am a
professor at the George Warren Brown School of Social
Work.
My areas of expertise are health disparities research
and implementation science.
Glad to be here today.
>> DR. BRUCE REED: Thanks.
Dr. Carpenter .
>> DR. MATTHEW CARPENTER: Hello, everyone.
Matt carpenter with the medical University of South
Carolina, my personal area of research is tobacco
control, smoking cessation, clinical trials and the
like.
>> DR. BRUCE REED: Dr. Gao >> DR. JINMING GAO: Good
afternoon, everybody.
This is Jim Ming Gao.
I'm a professor of pharmacology and oncology.
My research area is in cancer center medicine and nano
immuno oncology.
>> DR. CHRISTINE HENDON: My name is Christine Hendon.
My research is in optical tenography with applications
in surgical guidance.
>> DR. BRUCE REED: Thanks.
Dr. Janelsins.
>> DR. MICHELLE JANELSINS: Hi, everybody.
My name is Michelle Janelsins .
I'm a tenured associate professor at the university of
Rochester in Rochester, New York, at the department of
surgery and neuroscience.
I'm chief of our division in supportive care in cancer.
I'm a member at Wilmont research institute and my
research focuses on cognitive decline and cancer
survivorship.
>> DR. BRUCE REED: Thanks, Michelle.
Dr. Palermo.
>> DR. TONYA PALERMO: Hi.
I'm Tonya Palermo.
I run a lab focused on a couple of clinical and
behavior research on management and prevention of
chronic pain in children and adolescents.
>> DR. BRUCE REED: Dr. Peifer.
>> DR. MARK PEIFER: Hi there.
I'm mark Peifer.
I'm a professor of biology at UNC Chapel hill.
My lab focuses on the cell biology of cell adhesion,
the cytoskeleton and cell signaling.
>> DR. BRUCE REED: Great.
Dr. Rajaram.
>> DR. NARASIMHAN RAJARAM: I was muted.
My name is Narasimhan Rajaram.
I'm at the University of Arkansas in the Ozark
mountains, social professor of bioengineering.
My lab focuses on methods for looking at treatment
response in cancer.
>> DR. BRUCE REED: And Dr. Villa.
>> DR. ELIZABETH VILLA: My name is Elizabeth Villa. I
am an associate professor of molecular biology at the
university of California San Diego.
My lab focuses on developing technologies for using
cryo-electron microscopy .
>> DR. BRUCE REED: And we have four people joining us
as ad hoc members for this meeting.
Let me have them introduce themselves.
Dr. Anderson.
>> Dr. Karen Anderson: I am a member of the Mayo
comprehensive cancer center.
My research focuses on tumor immunology and molecular
diagnostics.
>> DR. BRUCE REED: Dr. Davidson:
>> Dr. Sean Davidson: From the department of pathology
at the university of since sings.
I study lack of protein metabolism.
>> DR. BRUCE REED: Thank you.
Dr. Spiekerkoetter
>> Dr. SPIEKERKOETTER: I am an associate professor in
medicine in the pulmonary critical care division at
Stanford university.
And my research is basic translational science in
pulmonary vascular disease including malformations and
right heart failure.
>> DR. BRUCE REED: Thanks.
And Dr. Yee.
>> Dr. Lynn yee: I'm an internal medicine in maternal
fetal science and my research focuses on health
services and health equity focused interventions for
pregnant people.
>> DR. BRUCE REED: And by no means least, but -- Noni.
>> DR. NONI BYRNES: Noni Byrnes, director and Center
for Scientific Review.
>> DR. BRUCE REED: Thanks, everybody.
So with that, we will turn to our agenda.
And Noni is up first with update on CSR.
Yep.
See the slides.
>> DR. NONI BYRNES: Excellent.
Good.
And you can hear me.
So this is our new setup.
The last couple of years, I have joined you from my
living room and here I am in my office.
I think it's our first day back for a subset of our
staff and we're starting to all come back, starting to
feel normal again.
So welcome, everyone.
Let me start off by welcoming our advisory council
members.
Welcome back to all of our continuing members.
And I want to introduce our newest members for whom
this is their first meeting as a full member. They've
been with us before as ad hocs.
Leopoldo Cabassa, Matthew Carpenter and
Christine Hendon.
So welcome to all three of them.
I also want to welcome our ad hocs for this meeting,
Karen Anderson, Sean Davidson, Edda Spiekerkoetter and
Lynn Yee.
Great to have you join us and this is a pretty dynamic
group.
And I know that you will participate fully in all of
our discussions here today.
So it's a full agenda.
As I do every time, I just want to get started with
what it is we do at CSR.
Talk about a strategic framework and our scope.
Just before every meeting, I think it's important for
us to be graded in that.
CSR is one of the 27 institutes and centers at the NIH
and we review applications for all 24 funding
institutes at the NIH as well as the office of the
director.
Our mission is singular and focused entirely on
ensuring that NIH applications get fair, independent
expert and timely reviews that are free from
inappropriate influences to allow the NIH to identify
and fund the most high impact and promising research.
It's an important mission.
I've talked about our strategic framework in the past.
So all of the members and even those who have ad hocked
before are familiar with this, but just a quick
overview, our framework, obviously, as our singular
mission is entirely focused on the quality of care
review and it's in three domains.
Study sections, whether the scope of the study sections
is appropriate to identify the best science, reviewers,
any aspects or qualifications and training.
And sort of inherent parts of our process.
All of these three domains very much impact what comes
out of the peer review process.
So it's an important part of our mission.
And we do this with the basis of our operating
principles.
And I hope that, you know, those of you who have been
here before and as you'll see today, that you'll see
that we're doing things with a basic core principles of
transparency with the public, using data to make our
decisions, engaging with stakeholders, especially
council, but all stakeholders and communications.
You've heard in the past different topics of discussion
related to different parts of this framework.
ENQUIRE is an ongoing process that you've heard about
before.
For reviewers, I've talked in the past couple of
councils, I had -- you know, I laid out strategies for
diversifying and broadening the reviewer pool and
shared some data related to that.
I've talked in the past about integrity and we've
rolled out integrity training modules as well as other
fairness bias mitigation.
So today's agenda will include, again, two
presentations on ENQUIRE by two of our division
directors, Ray Jacobson and Ross Shonat. In addition,
you will hear from Hope Cummings who will talk about
the survey analysis of the bias training that we
launched last fall.
And then Kristin Kramer, who directs our communications
and outreach effort is going to talk to you about her
office's scope and get council advice on some of our
outreach efforts.
Then, of course, again, last but not least, you are
going to hear from Elizabeth villa and Bruce Reed about
an interim report of the fellowship working group that
is looking at criteria and how those impact outcomes.
We have, if you recall last time, one of our division
directors discussed with you our draft strategic plan.
That plan was issued and -- as a draft. We are now in
a comment period.
I've actually, you know, really been impressed by all
of the comments that we've received from individuals,
from societies. They are really well thought out and
helpful to us. We have extended the comment period
until the end of April, April 30th.
And what are we going to do moving forward once we have
all the comments? We will do a content analysis, look
at all the feedback. We will adjust aspects of the
plan that we can and that we feel are necessary based
on the feedback. We will publish a summary of the main
feedback we got and then we hope, if all goes well, to
issue the final plan sometime in late spring of this
year.
Another slide that I like to show every time just to
remind people about the scope of CSR's operations and
just when people talk about changing things in peer
review, it's good to know how broad that is, how broad
implementing change can be.
It covers all of NIH science.
So NIH in fiscal year '21 received 88,000 applications.
Of those, 75% were handled by CSR, so that's around
66,000 applications in just one year.
That includes a majority of the RO1s and fellowships
that CSR is very familiar for.
We didn't do this alone, obviously.
This is with the invaluable assistance of greater than
20,000 reviewers, individual reviewers with whom we
engage and who did this entirely voluntarily. $200 is
pretty much a volunteer gig.
They did this to identify the best science in our areas
and we did this in around 1,300 meetings.
In addition to RO1s and SVIRs and fellowships, CSR in
the last fiscal year reviewed 182 special initiatives
of the NIH.
Many of these you will recognize.
These are just a few examples of some of the high
profile initiatives that our staff oversaw.
This is obviously not complete. We have many special
reviews that we handle whenever something is sort of
high profile, trans-NIH, it is CSR's responsibility to
handle those reviews.
This is a plot that I've shown the last few times
because I think a question came up maybe a few councils
ago about whether the application rates from women -- I
think the question was specifically related to women --
had fallen post pandemic.
They haven't for R01 submissions.
They've stay steady.
The overall submissions of women and by
underrepresented minorities.
This green line represents the pandemic.
So we're continuing examine this.
Last time I think I showed how scoring outcomes occur
for women post pandemic, as well.
I'm happy to share that data again.
But there wasn't that much of an impact in terms of the
outcomes of review.
And I think the question originally raised was about
success rates, which happens about a year later.
And I know that OER has published some information
recently on success rates.
Let me turn to CSR news, especially personnel news.
Then I'll update you on a couple of items that you had
asked for, initiatives that the council had been
involved in and what is the update on those.
So I'll go to those next.
So let me start with personnel.
With, you know, several leadership and management
transition that's we've had at the -- at CSR since last
time we met in September.
So the first item is a sadder one.
Not necessarily a promotion or a new person, but we
have our chief of basic neuroscience, Carole Jelsema,
is retiring at the end of next month, I believe.
And I wanted to acknowledge Carole's service.
She has been an integral part of the CSR community for
almost three decades.
She's been involved in neuroscience reviews throughout
our career and she's led a lot of high profile
initiatives and trained an entire generation of SROs.
So she will really be missed and I want to wish her
happy retirement out here in public with the council
here.
We created a new position called senior SROs.
These are -- this was specifically targeted to create a
cohort of SROs who are sciencely flexible.
Suddenly initiatives come up and they're high priority
and they're fast track.
This allows us some flexible.
They cover everything from basic science to
epidemiology, social science, bioengineering,
everything.
So it's a really nice group that we have and I want to
acknowledge John Arias, Jessica bellinger, Michael
Bloom, Heidi Friedman, James Li.
They're located in the office of the director.
We have three new branch chiefs.
They oversee SROs and study sections in given
scientific areas.
So Tom Beres is the new chief of the health services
and systems branch.
Emily Foley is the new chief of disease control and
applied immunology and elia Ortenberg is head of the
social and community influences across the lifecourse.
In addition, Dipak Bhattacharyya who was our acting
director is now the new director of a new division of
planning analysis and information management. Because
of our focus on data-driven decision making -- and
you've seen a lot of our analysis and data come out of
CSR, this has been an integral division for us and it
was necessary to create a broader structure to manage
all the work going on in analysis and planning.
So please welcome Dipak to the leadership team, CSR.
And then if you recall, last time I talked about the
establishment of an office of training and development
for SRO and reviewer training within the office of the
director. And I had announced that Vanessa Boyce is
handling the new SRO training.
So these coordinator positions -- Vanessa is one of
them -- are full time SROs.
And they have one element of their job.
So part-time they assist in areas of their expertise.
So Vanessa oversees the training of our new SROs.
I am happy to announce we have completely staffed up in
our office starting with Miriam Mintzer, director.
And we have three new coordinators this time. Tanya
Cohen, overseeing reviewer training, Ben Shapero is
overseeing the handbook and policy coordination and
Natalia Komissarova is overseeing continuing education
for all of on our SROs.
So I'm happy to have this established and have a
rigorous training program under way at CSR.
You had wanted, I know, to hear about an update on the
early career reviewer program.
Just a reminder for those who were not here when this
started, back in 2019, an advisory council working
group looked at the ECR program and rehauled it. You
know, and part of the main change that was made --
there were many and multiple changes made -- but the
main change made was to double the number of ECRs that
we were using in each study section.
They also developed new criteria to really target the
program more towards where it may have the biggest
impact, which is career investigators who don't have
review experience or an R01. So it positions them to
experience the new process at a time when it can have a
big impact.
And instead of having individual SROs select ECRs and
evaluate whether any CR is qualified or not, which
leads to some discrepancy -- of course, we have 250
SROs.
So they made this a central process. We have an SRO
committee that looks at them and looks at the criteria
for acceptance into the program.
And we had a new database because the past database did
not allow easy tracking or evaluation.
So going forward, we are able to track.
So those were the changes made.
Let me give you a quick update on what our early career
reviewers sort of the demographics of the early career
reviewers looked like.
We just compared the February/March meetings since the
program was rehauled. You can look that females, among
the cohort were increased.
That's from 49 to 55% almost.
Under represented minority, it was already kind of
enriched and went up significantly in the next year and
then kind of fell again this year.
Talk a little bit about why that might be.
And the same trend is true for black African-American
reviewers and to some extent for Hispanic reviewers.
So we can speculate.
We haven't done a deeper analysis yet to see what
happened and why that bump happened.
I suspect -- and Kristin can fill this in when she
gives her talk later today -- about communications is
that we did a pretty successful webinar outreach to
recruit early career reviewers.
And I believe that that bump occurred soon after that.
So it's possible that we need to do more of that kind
of outreach, but we have to do a lot more analysis
first.
The other thing that we noticed is that different
scientific areas are represented differentially in ECR.
So some have more ECRs and are oversubscribed and some
have a lot fewer.
So we are targeting our efforts towards scientific
areas where we don't have as many ECRs signing up.
And then one of the questions that comes up is the
outcomes for ECR.
So this is sort of program outcomes starting in 2012
when the program was first initiated.
And, you know, 46% or so have competed for an R01
successfully, which is good.
And around 14% have served as standing members.
These data are little -- there's no real control for
these, right?
So we don't know if they would have done it despite or,
you know, whether the ECR program made a difference.
It's hard to know that and this is not a good -- we
haven't quite gotten our head wrapped around how to do
this analysis in a way where we can see the effect
specifically of the program.
So we plan to continue targeted outreach efforts both
broadly as well as targeted areas of science and
further analysis, as well.
The other update is related to the simplifying review
criteria effort. Several of you were involved in the
working group. There was a non clinical trial working
group that was convened in early 2020.
And then there was a clinical trial working group with
several of the same people.
Tonya was the co-chair in both cases.
That presented its final recommendations to this
advisory council and got approval for those, after
which it moved to NIH for further development.
So just an update, in July of 2021, this was -- this
concept was cleared and approved by a senior leadership
committee at NIH which looks at everything extramural.
They recommended we form a working group to see which
ones are feasible and practical.
The working group met. I was a co-chair of the working
group along with John Lorsh.
We did the same thing that your working group did,
which was break it up into non clinical trials.
Clinical trials is a little less than 10% of the
applications and it's ten times more complicated.
So we thought we would do this first.
So we developed recommendations and presented those
back to the extramural working group where it got
approval.
And the next up is next month we will be presenting
these recommendations to the NIH steering committee.
So just stay tuned. It is moving along through the NIH.
The other thing is not really an update, but just a
reminder.
Gabriel Fosu is our associate director Of Diversity and
Workforce Development.
Last month, CSR established a reporting avenue, a
direct way in which investigators and reviewers can
report by us and review to CSR.
We include it in our signature as we tried to publicize
it.
It's gotten a little bit of traction, but not as much
as I thought it would. We have some Emails coming
through, but not a lot.
So we welcome your assistance in getting the word out
that there is a way to report.
Every allegation is investigated by CSR senior
management.
Both Gabriel and the scientific division director
together look at every allegation carefully.
If we agree that this was a bias review, we review it
because we don't want to disadvantage the applicant.
If we don't agree, we guide them to go through the
normal appeals process that is always available.
One point that I wanted to make, which is something we
didn't used to do before. We have done rereviews in
past years, as well.
But we never look back to the reviewer who caused the
problem, right, with the statement that was bias. We
are doing that regularly now.
So the division director calls the reviewer, tells him
or her, you know, this is what you did and this is the
problem that it caused.
It caused us to rereview.
I think that's important for reviewer education, for
culture change, for them to know what caused an issue.
And then we watch to see what kind of reaction.
If the reaction is oh, wow, I've messed up, I've
learned something, that's a good thing.
If the reaction is that is not a bias statement, that
gives us more information about the reviewer and we
take further action as necessary.
All right.
And then finally, let's talk about the future of CSR
peer review meetings.
You've seen this before.
I'll go over there very quickly. We have the reports
there, but clearly, one of the main areas of concern
with Zoom meetings -- overall, we're doing fine.
Overall, quality seems okay.
There is many types of evaluations that were done and
overall it seems like the outcomes of review are okay.
They're working fine in a virtual setting.
Reviewer engagement suffers.
I think that is not surprising.
It's true for anything that is happening through Zoom.
Because of that, you see also a similar worsening in
the productive discussion.
This is reviewer impressions of the meeting.
And it's pretty consistent.
We did this for two years in 2020 and 2021.
I showed this last time. Which is kind of still
interesting, which is the preference.
So most reviewers prefer in-person meetings.
Some prefer Zoom meetings and some have no preference.
When you break it down by gender, both men and women
prefer in-person, but the difference is greater for
men.
So that is an interesting thought and we can come up
with reasons as to why that might be.
We did this for career stage and we also went back and
did it for chairs, which kind of maps on to the full
professor outcome. Which is, again, all career stages
prefer in-person, but the margin is larger for full
professors for more senior reviewers. We can talk a
little bit why that might be, as well.
I've shown this before.
We've done a pretty comprehensive analysis of score
distributions and out of range scoring.
So actual outcomes, not just reviewer impressions of
meetings.
There is not a real difference in the score
distributions from in-person versus Zoom taking the
same study sections.
So in the end, we've talked about this before, both
formats have their own strengths and weaknesses.
For in-person meetings, we know more reviewers like
those.
They're more engaging. You're more attentive.
There's a chance for the group to come together and
form some ties and bonds.
It's really important for recurring panels.
And people want to come because they get to talk to
other people.
That is lost during zoom.
Negative, of course, is all the time that is spent
going back and forth in planes and Ubers.
There are other aspects that might make it more
difficult for others to participate, child care,
clinical, our types of duties.
For us, this is important because it might hurt
recruitment efforts and broader participation that
we're interested in seeing.
And then, of course, the environmental and fiscal
impact.
For virtual meetings, some reviewers, a good 30% or so
prefer virtual meetings.
The time saved is hard to compete with.
It's a lot of time saved and still participate and
review.
It's easier for people with other duties.
And, of course, there's no environmental impact and
fiscally it comes out really well for NIH.
But the negative, again, what we've heard, reviewer
engagement, attention span, no chance for the group to
come together and then you don't have the networking.
So there is a balance.
And so in fall 2022, we will start to hold some
in-person meetings beginning in the fall. There are
people -- because all the mask mandates, even in the
really blue virulent are gone.
Many people are saying why don't you have meetings this
summer in person?
This is when you have to look back at the scope slide I
showed you. 1300 meetings, 20,000 reviewers.
It's not that easy to do quickly.
We need five to six-month lead time to do all the
contracts and all the travel.
So what we're going to do is provided everything stays
stable in terms of COVID, all standing panels, those
that have recurring membership or chartered study
sections, SVIR and fellowships will hold one in-person
meetings a year.
And about a third of our panels will meet in person
this fall, ideally, and the remaining two-thirds will
hold in-person meetings coming up in the next two
rounds.
This will all depend on reviewer preference, especially
member preference.
So if we lose half the members because they don't want
to travel yet, they might be more comfortable next
June.
Those are probably the ones we'll delay a bit.
We're going to go slowly.
We want to make sure that we survey our reviewers and
SROs to see what they're comfortable doing this fall
versus in the spring and then the summer.
And for me, it's really important that we're looking at
recruitment success, are we able to recruit as broadly,
are we able to get women and clinicians to support in
in-person meetings.
We have made a decision to not hold hybrid meetings
this fall.
As we go out of the gate, we're not going to do hybrid.
Hybrid means some people participate virtually and
others are in the room.
Part of it is technological.
We don't have as much technological capacity. Hotels
and all of that, it's more difficult.
But, really, it's more about ensuring that our staff
and all of us understand the issues related to inequity
of participation and experience with the virtual people
and learn how to manage that.
That my change in the future so we're open to that.
But just this first go round, we're not going to have
hybrid meetings.
That said, the September 2022 meeting of CSR advisory
council will be held in person.
The date has slightly changed because the first -- the
lasts Monday, I believe, is a Jewish holiday.
So this is the last Monday before the Jewish holiday.
So mark that down, all of the members, and we hope to
see you here in Bethesda.
Now, I want to end with an acknowledgement.
March 2020 is when we all went to virtual meetings and
remote work.
And I want to acknowledge our staff because for the
last two years, you know, the really important work of
scientific peer review, that is so critical to NIH
funded research continuing.
It has continued uninterrupted thanks to our staff, our
scientific support administrative and technological
staff have all worked together, and at a time when the
pandemic was disrupting their drives just as much as
it's disrupting all of the scientific community's life
out there.
This is what we've been doing for the last couple of
years, but we're hoping that we can get back to that
building on the left as where Bruce and I are sitting
right now. We have a couple of buildings.
And then we have some remote staff across the U.S.,
which is interesting and fun.
So thank you to CSR staff.
That's it for me.
>> DR. BRUCE REED: Thanks, Noni.
So it's open for discussion.
Anyone?
Jinming.
>> DR. JINMING GAO: Thank you for the update on the
review progress related to consolidation. I'm thinking
that it seems like it's moving along, I guess, nicely
and with the council review. I was thinking about with
the in-person meeting coming up, how would that change
with the review process being implemented?
Would that -- where a timeline first and then the
implementation, I guess, if it is approved by the
council?
>> DR. NONI BYRNES: So I guess I'm not clear, Jinming,
that it would make a difference. I think the review
criteria -- that's what you're talking about, right?
>> DR. JINMING GAO: Right.
>> DR. NONI BYRNES: If that were to move forward, and
I can't say that it will. We have to take it through
the steering committee and beyond.
But I suspect something will move forward.
It's not going to happen this year.
Like I said, it's going to take another year or so just
to get our systems, just ERA itself takes a long time.
And I hope we will be very familiar with handling
virtual and in-person and hybrid meetings by the time
that comes around.
>> DR. JINMING GAO: Yeah.
That's where I was coming from. I was worried about
too many things going on and --
>> DR. NONI BYRNES: Yeah, no, I understand.
>> DR. JINMING GAO: Yeah.
>> DR. BRUCE REED: We wish, but it won't change that
fast.
Lynn.
>> Lynn Yee: Thank you, Noni. That was a great
overview.
My question is about the ECR reviewers.
I really -- you know, as one of the more junior people
on this panel, maybe one of the few who participated
and I did the program four or five years ago and found
it incredibly informative.
What I was wondering about after seeing the statistics
on who participates is about the statistics on people
who apply that are never selected and how -- you
know, what their characteristics might be and if they
might be a good comparison group for determining the
outcomes of the ECR program.
That is the people who apply, but are not chosen, but
still, you know, what proportion of them both apply for
an R01 and receive an R01
>> DR. NONI BYRNES: That is a good point.
Haven't -- one of the things we have yet to do is go
back to our database and do a rigorous analysis of who
applied and who got in and how they're doing,
especially with areas of science.
So we plan to present a future update to council, maybe
in a year or so, a couple of more councils and have
more information. That would be good to see, you know,
who we didn't select.
But the criteria is so clean, right?
I mean, you know that -- this group developed the
criteria and they're very clean and clear.
>> Lynn Yee: I agree, there are. I know of people who
said I applied and I never heard back.
And I wonder -- I think sometimes early career
investigators might like feedback on why they aren't
selected. I know that would be a large undertaking for
CSR to provide that feedback.
>> DR. NONI BYRNES: Is this recent experience?
>> Lynn Yee: No.
It was a few years.
>> DR. NONI BYRNES: That's one of the reasons we did
this review.
We're trying to do that more in the last couple of
years.
And I think they do hear back if they don't get in.
>> DR. BRUCE REED: They certainly hear if they
didn't -- if they didn't get in.
Turnover is pretty brisk in that program. I don't
know -- I'm sure there are some who stay in the pool
and not selected, but my concern has been more along
the lines of keeping enough people coming in to meet
the need and study sections.
We use about 380, about 380 people around and we
have -- I don't know, Kristin, we've got somewhere
maybe around 1,000 people in the pool.
So --
>> DR. KRISTIN KRAMER yeah.
So let me expand a bit on that.
So we have about 1200 people in our database.
And it's kind of hovering around 1200.
And in terms of the wait time, it's much reduced from
what it had been before the working group took this on
and established new criteria for participation.
Right now, people who applied and were accepted into
the program in the last two years, 75% of them have
served on a panel. So the vast majority of people who
are applying are serving on a panel.
The wait time is relatively little.
But we have that 25, almost 25% who haven't served yet.
And I think that's largely due to the fact that there
is a very large oversubscription of early career
reviewers in certain fields.
So, for example, there are some areas such as in social
sciences, population sciences, where we have as many as
300 early career reviewers in our database for those
study sections in that area. When we're only recrewing
two early career reviewers per standing panel, there's
a lot of people who are waiting and who probably will
not get the opportunity to serve as are reviewer before
they're tenured.
So I think that's where the large wait time is.
It refers to oversubscription in certain fields.
>> DR. BRUCE REED: Thanks.
Mark.
>> DR. MARK PEIFER: Thanks for that update on the ECR
program.
You guys have made remarkable progress on that.
And as somebody who played a small role in that, that's
really -- that's nice.
And just another kudos, right, for continuing to follow
the impact of the pandemic.
And I'm confident you will continue to do that and
continue to report back to us.
Again, I'm surprised but pleased to see the application
and thus far taken a big hit but, again, I think it
will be important to follow that out for several years.
But thank you for doing that.
>> DR. BRUCE REED: Thanks, Mark.
Edda.
>> Dr. SPIEKERKOETTER: I had a question reporting bias
in the CSR. Do you have any more data on how many PIs
form kind of this -- or, like, contact Gabriel, how
many other reviewers, who is the outcome? Because I
feel often people feel they have applied and that maybe
found some kind of bias feel a little bit helpless that
they don't really know where to turn to and whether
they will be successful and whether they are hurt and
all this.
So do you have a little bit more information on that?
>> DR. NONI BYRNES: Yeah. Unfortunately, we haven't
had a lot of traffic into that mailbox since we -- we
established it a year ago in March of 2021. We have,
however, gotten more bias reporting or other concerns
raised through the SRO or through our chiefs or
management, not through the box, necessarily.
We are getting a few.
I don't have the numbers yet. I would simply -- I
don't know if anyone -- I don't know if Gabriel is on,
but it's very small. I would say a dozen to 12 to 15
that we've actually done anything or gotten any
significant action on.
I want to be -- that's why I'm trying to publicize it
more. We will do it through social media and others,
but we would appreciate your help. We have taken
action and when I have some more numbers, hopefully in
a future council meeting, I'll report the specific
types of actions.
We have other division directors here who can tell you
anecdotally, but we don't have numbers yet.
>> DR. BRUCE REED: Thanks.
Matt.
>> DR. MATTHEW CARPENTER: Thank you so much.
Wonderful presentation.
I think I saw or heard the statistic that there are
20,000 reviewers.
And I was wondering what kind of data you keep on the
quality of these reviewers. I know when I served and
chaired study sections, the vast majority of reviewers
were outstanding. There was a small minority who were
not.
And I'm wondering how you keep track of us.
>> DR. NONI BYRNES: Okay.
I'll invite anybody to jump in, but let me just tell
you that we don't have a central way of keeping track
of 20,000 reviewers.
We have SROs who are very active and will not invite
back people who are not strong. We are -- I do think
as we bring -- as we broaden the pool of reviewers that
we're actively doing and bringing new people into the
system, it's really important to have strong reviewer
training and reviewer feedback loop.
And I think that's what our office of training is
targeting -- is trying to target to do that in a much
more systemic way.
>> DR. BRUCE REED: Yeah.
A couple of thoughts. You know, so you can
differentiate between reviewers who are probably not so
good and those who are, shall we say, like integrity
challenged in that the integrity challenged ones, we
act vigorously to block from further review service.
But the other question, as Noni said, we've really just
begun to think about this systemically.
So how could we really evaluate reviewers at scale?
As you say, 20,000 people say how do you do this in a
way that would provide us with useful feedback.
And it's not simply a matter of trying to screen out
good ones and bad ones.
It's try to link it to reviewer training and trying to
identify, for example, scoring behavior that might
benefit from additional training or those kind of
things.
So we don't have it worked out, but it is on our agenda
to give that much more careful thought about how we
should be, you know, as you say, tracking, but
evaluating reviewers with an eye wards improving
performance overall.
>> DR. NONI BYRNES: And let me follow up really
quickly, Matthew, is that the strategic framework that
I showed, reviewer evaluation is very much a part of
that.
But you didn't see that highlighted and that's because
as Bruce said, we're just starting those efforts and it
is parts of our strategic plan, as well.
>> DR. BRUCE REED: Tonya.
>> DR. TONYA PALERMO: Thanks sort so much for the
update.
I appreciate the efforts contributed over the last few
years.
I had a question about the bias sort of reporting and
wondered if you have gotten feedback from the
community and from applicants specifically about what
they think warrants, you know, an email to Gabriel.
Are they conceptualizing only the most egregious forms
of biases fitting that or do they see the spectrum and
understand that that is available to them.
My gut is they think it's for egregious forms of bias
and that they're probably not understanding the full
spectrum of what we're hoping for there.
But I'm curious if you've gotten some feedback from the
community about that.
>> DR. NONI BYRNES: We have not.
I do know that what we have gotten, that we've taken
action on has been pretty egregious and pretty obvious.
You know, I'm sure others can weigh in.
I want to invite any of the division directors to
unmute and join in because they're the ones who really
handle this with Gabriel.
But, yeah, it's been pretty egregious. I don't think
we've had a lot of subtle bias things submitted.
>> DR. BRUCE REED: One thing I want to make sure
people are clear on, anybody who has a concern about
bias, there's multiple avenues through which they can
report. You know, so the bias mailbox is one, but we
actually get a lot of reports directly to the SROs or
to the chief or to the division director.
Those are perfectly fine.
We absolutely act on them that way, too.
So we get them through a variety of channels.
>> DR. NONI BYRNES: And we're actually tracking all of
that centrally, as well.
So that in the future, I can report sort of the scope
and scale of what we got and the types of reports we
got.
>> DR. BRUCE REED: Michelle.
>> DR. MICHELLE JANELSINS: Hi, Noni.
Great updates all around.
I have a question about the logistics for the
in-person, going back to in-person meetings and Seth up
to the one in-person meeting a year. I was wondering
if you're going to give flexibility to study sections
or if they have a meeting in September in-person, will
it be anticipated next September they would also meet
in-person or is there flexibility given or will it be
random?
So sometimes will study section might meet in the
winter one year or it might meet in the spring or
summer the following year.
I'm wondering if you can comment on that.
Also if you've heard kind of any thoughts about people
avoiding winter months or any context like that.
Thanks.
>> DR. NONI BYRNES: So thanks, Michelle. I will say
yes, we've heard thoughts about avoiding the winter
months, absolutely.
So I'll answer that.
And then I'll turn it to Bruce because he's been --
we're just starting to grapple with this and figuring
out how we're going to deal with this.
And I believe we're going to start with surveys with a
lot of flexibility.
Bruce.
>> DR. BRUCE REED: CRS will be surveying both
reviewers and SROs looking at their preferences.
And I have to say, Michelle, your question is well,
you're a step ahead.
We're just trying to get some meetings into place here
and trying to look at a year at a time.
Ultimately, we do have to figure out those questions
that you asked.
We have heard -- it's a fairly common concern about the
winter months. You know, weather problems aren't
unique to the winter, but we run into more of them.
Also you can think public health issues might be
greater in the winter than other seasons.
At this point, we're trying to space out this first
batch and this is very much a learning as we go
process.
But thanks.
Christine.
>> DR. CHRISTINE HENDON: Hi, Noni. Thank you for the
overview.
I had a follow-up question regarding logistics for
in-person meetings. Will the in-person meetings
primarily be in the D.C./Maryland area or will they
rotate around the country?
And then the other was prepandemic, we've always had
some reviewers participate, you know, via phone.
Will that continue?
>> DR. NONI BYRNES: So I think we haven't made any
decision about only he holding these in the D.C. area.
I suspect that we will allow meetings elsewhere.
We haven't quite worked out a hotel contracting and all
of that is now 3-year-old information.
Not even sure those contracts are in place.
And then participation by phone, yeah, we will allow
that as we always have.
And, you know, the hybrid option is available in the
future. We know how to do Zoom pretty well.
We just need to work out how to management equity of
participation.
Just this first round is when we're not doing it.
>> DR. BRUCE REED: And Elizabeth.
>> DR. ELIZABETH VILLA: Hi. Noni, this is a little
bit out of left field because it's not related to this.
But thank you for following up on the women. I am
still not surprised, but very happy that women continue
to do well regardless.
That doesn't mean that it wasn't awful.
But I also want to ask about -- we didn't talk about
this and I didn't ask you before, but what is the
status for ESIs with the pandemic? Because it's not
done.
And so how will their status might be extended upon
request or anything if they can -- you know, if they
ask for an extension or something due to care taking
responsibilities or the like during the pandemic?
>> DR. NONI BYRNES: So I don't know.
I'll have Bruce and Kristin if they have heard anything
to the contrary.
This is outside of my lane.
>> DR. ELIZABETH VILLA: I realize.
>> DR. NONI BYRNES: I'm just not aware.
But I'm happy to touch back with OER and follow up with
this group.
>> DR. ELIZABETH VILLA: Thanks.
>> DR. NONI BYRNES: And to just ask him to publicize
whatever they're planning --
>> DR. RAY JACOBSON: Also, I know this is not CSR
business, but since this is an official channel, I
think in general the CSI status might need some
revision from NIH based on when they started their
position rather than when they finished or terminal
degree or other things.
>> DR. NONI BYRNES: Noted.
>> DR. ELIZABETH VILLA: I know it's not CSR
>> DR.
NONI BYRNES: I'm happy to pass along.
We've done that before for this group.
>> DR. ELIZABETH VILLA: Thank you.
>> DR. BRUCE REED: I just got a note from -- to the
effect saying ES extensions are still in place. I
can't tell you what the details are on that, but I
think OER is aware of the issue, anyway.
And we will try to get you more information.
>> DR. ELIZABETH VILLA: I found some interesting where
it says you can't get both an extension for COVID and
having a child, you know, because young parents won't
get affected by the pandemic somehow.
But, yeah, some interesting discussions there.
It would be very nice to pass along to whoever needs to
hear it. Thank you.
>> DR. NONI BYRNES: I will.
>> Lynn Yee: One thing to add to that topic, the
physician science have a different experience in the
pandemic, researchers who were pulled away from
research to provide care in the hospital might be
another reason to consider how to modify the ESI
status.
>> DR. ELIZABETH VILLA: Absolutely.
>> DR. BRUCE REED: That's a very good point.
I'm going to take advantage to move to the next item on
the agenda because we're right on time here.
So thanks, everybody, for that discussion.
The next item up here is introduction to the CSR
communications office.
This is going to be a presentation by Kristin Kramer.
So I will turn this over to Kristin.
>> DR. KRISTIN KRAMER: All right. Thank you very much.
I appreciate it, Bruce.
Let me share my slides.
>> DR. NONI BYRNES: Yep. I see them.
El >> DR. KRISTIN KRAMER: So today I want to give you
an update on what the communications and outreach
initiatives office is doing.
Our broad goals are to increase transparency and
facilitate stakeholder input. There are a number of
things we hope to gain by doing this.
First, we hope to level the playing field by equal
access to information.
When I say this, I'm thinking about the fact that some
investigators have access essentially to an informal
network like colleagues in their department, in their
hallway who have NIH funding and experience surveying
on NIH review panels.
But not everyone has that network to draw upon.
So by increasing outreach to ensure that investigators
understand the review process, we can potentially
decrease the importance of coming center from a network
or institution that has NIH experience.
By increasing transparency in terms of around our
policies and procedures, we can increase trust in the
process and in CSR and NIH. We can capitalize on
stakeholder input. We can gain from their perspective
and experiences to make improvements, be it in the peer
review process or even in our own workplace culture.
So our audiences are Congressional staff, external
scientific community, those within NIH but outside of
CSR and our own staff inside of CSR.
Each of these audiences can be further subdivided.
Example, the external scientific community, we might
think about them in terms of investigators, reviewers
and office of sponsored research.
So today I'm going to focus our work on trying to reach
the external scientific community.
We use a range of communication tools to reach these
audiences.
Social media, Twitter and LinkedIn.
We give presentation that's are aimed at large
audiences.
In the past, we have handled facilitated discussions
with the external community.
Then this last one, direct email campaigns and direct
replies, this is simple and low tech and time
consuming, but I think this is one of the most
important things my office does.
We respond to every email we get and I think that
one-on-one interaction is important for building trust.
It gives us a lot of detailed information we can use in
terms of understanding the audience's needs.
So first I'm going to talk a little bit about review
matters.
This is our blog.
These three posts are our top posts in terms of page
views.
Raise and peer review was posted by Noni Byrnes in
2020.
Should we keep meeting this way.
And then our current post that is up calls for comment
on our strategic plan.
So you can get a sense hereof on our typical page
views.
The top one has about 9,000 page views.
Since the beginning of March.
That's the cumulative page views.
When we look at the analytics, sometimes the office
cross posts our blog so we see another increase and
then a gradual increase in additional page views over
time.
The blog is valuable for allowing us to get comments
and feedback from the external community.
So we typically publish comments that are directly
relevant to the post that we made, but we receive many
comments that are tangental.
If we don't publish those tangental comments, we read
them.
We're fortunate is to have a great team that helps us
collect those comments and synthesize them in a way
that we can take action on them.
This blog has about 121,000 subscribers.
That clearly does not translate into page views.
So there is room for improvement.
We probably need to kind to have clean up our
subscription list and think about a different way to
develop a subscriber base.
The comments often suggest we're reaching a different
audience with the blog versus the audience we reach
through Twitter. For example, many of the comments
refer to a long history with NIH.
To me, that suggests the importance of using multiple
means of communications and multiple social media
platforms. Because even though I subdivided our
external community audience into investigators and
reviewers and offices of responsive research, some of
those offices can be further subdivided.
So we might think more about how we use social media
and diversify so we can reach all of our audiences.
We need to use analytics to better understanding who
our blog is reaching and to increase our reach of this
blog.
So Twitter can be used for a number of things.
It's obviously useful for sharing information. So the
tweet on the right is just a tweet about our new chair
orientation that we do every year.
So when we train our chairs, we publish a video of that
training to be transparent so that the external
committee knows what advice we're giving to our chairs
and how we respect meetings to be run.
The tweet on the left seems minor, but it happens to
away tweet that had a lot of eyes on it.
And this came about just because I saw on Twitter that
there was a lot of concern about having a figure such
as a flow chart and specific aims.
Investigators had found that they were getting warnings
when they submitted grants with a figure on specific
aims.
Even though ERA make public all the history around
warnings and errors and what triggers them, it's
difficult for the external community to find that
information.
So I was able to find that end tweet and provide some
assurance to investigators that their grants would
still move forward with review even if it had a figure
in the specific aims.
And I think we can work on this so that we understand
what the pain points are each round, what questions are
they getting repeatedly and we can kind of head it off
and start communications in advance of grant deadlines
to reduce some of those problems.
We can use the input that we get through Twitter to
improve our processes.
So this tweet is from someone who really likes the Zoom
format, but pointed out that the meetings he had
participated in had gone really long. 14 to 16 hours.
And we heard this from reviewers through other means,
as well.
So in response to this, we issued guidance to SROs to
remind them to keep the review meetings to a normal
length of day instead of trying to take advantage of
the fact that people didn't necessarily need travel
time.
I think we can use Twitter to instill trust.
So this tweet came about because an investigator
tweeted about a grant that is withdrawn.
She felt we made a mistake.
So we reached out to her directly, we looked at the
details and we realized that we had, in fact, made an
error.
We felt that it was important to tweet in response to
her tweet and make sure that people in the community
understood that we know we're not infallible, that we
do make mistakes and when we find out that we made a
mistake, we do everything we can to address it in a
timely fashion.
We also wanted to make sure the external community knew
there was an appeals process and they could contact the
division of receipt and referral directly, that they
didn't have to rely on making me see something on
Twitter.
So this tweet came about because I saw a conversation
on Twitter about bias and NIH funding and bias and
review.
Dr. Watson wasn't necessarily aiming his tweet at CSR
or asking for action, but I took the time to read
through his summary statements.
I admit that I didn't fully understand what the issues
were. So I contacted him via email and we had a
lengthy exchange.
I think it was really helpful to basically sharpen my
eye for bias.
And I think it was helpful in terms of thinking about
the content and the bias awareness training.
One of the things that was great about this, this is
before we had a reporting avenue for bias.
He tweeted that, a, we were receptive, so that is
great.
He also pointed out people in the community could reach
out and talk to their SRO if they were concerned about
bias and review.
So our websites also provide a great way to give
detailed information to our communities.
Our current website is set up with the applicant
community in mind for reviewers.
And then there's other high level menu items for news
and policy and study sections and the like.
So each of these connections has more content developed
specifically for that audience in mind.
So applicant's channel has information on the
application process, planning and writing, submission
and assignment and so forth.
When we look at the analytics, even though we invest a
lot of time in those areas, that's not where people go.
So I want to call your attention to the top bar in
darker blue.
These are our page views for the month of February.
All of our CSR pages combined had a total of about 210
page views. When we look at the rest of the content on
the site, what we see is that study sections -- these
are the description pages for each study section -- has
far and away the most page views.
So study section description pages account for about
120,000 of those 210,000 page views.
And the other thing we can see from analytics is that
people are not going to our landing page and browsing
to get to the study section pages.
So we spend time thinking about the content that is on
that landing page, but actually relatively few people
see it.
People use Google.
They Google an acronym for a study section and they go
directly to that study section page.
So what that means to us is study section description
landing pages are cues.
We need to use those study section pages thoughtfully.
So we have long had a reporting avenue for breaches in
integrity.
But as Noni mentioned, in the last year, we established
another reporting avenue for concerns about an unfair
or bias review.
When we did this, we added a bar on every study section
page so that people had ready access to information
about how to report these concerns.
We can take this a step further and we can think about
how to thoughtfully drive traffic to other parts of our
website.
So this is a screen shot from an SRO add.
In the month of February, we had about 3,000 page
views.
So in thinking about the value of study section
description pages, we added banners to all the study
section description pages that would point to this
advertisement. When we look at the data, we see that
almost 1300 page views of that add were referred from
study section pages. In other words, people were on a
study section page and they went from that study
section page to our advertisement.
We can also use analytics to see the impact of
different forms of outreach.
So these -- this is a tweet and a post on LinkedIn,
again, about an SRO advertisement.
And on the right, you can see the number of page views
over time.
And each of those spikes correspond to some effort in
outreach.
So we can do a little -- what we can do in the future
is do better analysis. We can try to reach out via one
social media stream and see what the impact is and take
a little bit of time, space it, put some time between
that tweet, for example, and a post on LinkedIn and see
how the page views change.
We've also done webinars and panel discussions.
This is an Early Career Reviewer webinar that we did
about three years ago.
This was a suggestion, we were discussing how we should
go about increasing the number of underrepresented
minorities participating in the early reviewer program.
Her comment was generally it's best to find a way to
let minority scientists see themselves in that role.
So we decided to run a panel discussion.
We publicized it through a range of means.
Dr. Gabriel Fosu sent information about this webinar
through networks that he's developed.
And I also reached out to offices Of Sponsored Research
and department chairs and minority-serving institutions
and asked them to share this information with their
faculty.
Finally, we used Twitter, as well.
What we saw was really interesting.
We did each of these outreach efforts at different
points in time. We could look at when we did the
outreach.
So, for example, when an email was sent to those 1,000
ESIs and compared it to registrations to participate in
the early career reviewer program.
What we found from looking at those data is that
Twitter was actually the most successful means to drive
traffic to the early career reviewer program.
So our webinar results were great. We had over 700
attendees, over 150 questions.
We saw an increase in registration in the early career
reviewer program, often without registering for the
webinar, itself.
And we saw an increase in early career reviewers who
were from groups underrepresented by medical devices.
So the data on the right side are from may of 2021 when
we were publicizing this webinar.
And the data in the left column are from the same time
span a year prior.
What you can see is that we had a large increase in the
number of people accepted to participate in the early
career reviewer program as well as the number of
underrepresented minorities who submitted applications
and were accepted.
So I'm going to briefly touch on some of the work we're
doing in communications within NIH.
Our broad objective is to ensure there's a better
understanding of peer review.
I'm thinking about what works well in peer review and
what doesn't and how we can ensure useful review
outcomes.
What CSR has done in that space is established review
liaisons to ensure useful review outcomes. We also
know, of course, that program officers spend a lot of
time talking to investigators.
And we want to make sure people within NIH are aware of
CSR initiatives and what our policies and procedures
are so they can convey that.
To that end, we've developed a new letter for program
staff and develop a web page so program staff can
communicate that investigators.
We also want to partner with ICs in the future so we
can collaborate on outreach efforts.
In many cases, funding institutes are developing
workshops at meetings of scientific societies and
instead of CSR going it alone and having our own
separate talk, it would be probably have a greater
impact if we participated alongside funding ICs and
added information on the peer review process.
And then another objective is to continue increasing
the diversity of our review panels and to collaborate
with those inside NIH to do that.
So one tool that we've developed to that end is
reviewer recommendation tool through which program
staff can recommend strong scientists to serve as
reviewers.
So generally I think this is an area we need to spend a
lot more time on.
Within CSR, in terms of our communications with our own
staff, I think it's important to remember that our 250
SROs spend a lot of time talking to reviewers.
So I think we need to expand in terms of providing
resources and materials to SROs.
In terms of engaging with our own staff, I think it's
important to connect all of our staff to CSR's mission
and to engage our staff in terms of getting input
related to policy and practice.
Also, I think we have an important role to play in
terms of improving workplace culture in terms of
diversity, equity, inclusion, accessibility effort,
hiring transparency, career development and staff
recognition. So I'll touch on that first bullet.
This is an example of a resource that we developed for
SROs.
So pretty early on in the pandemic, we heard from
reviewers that there was some concern that SROs maybe
weren't being empathetic to the level of stress that
reviewers were feeling.
So we issued guidance to SROs, which included things
such as inviting reviewers to let them know if they
needed reasonable accommodations.
We asked SROs to explain their thinking to reviewers in
terms of setting the meeting date.
We asked, SROs to review the meetings nodes. SROs have
a welcome letter that they send out to reviewers and
communications they send out in advance of critique
deadlines. In a normal world, we recycle dates and
deadlines.
I think in the pandemic world, we were continuing to do
that.
So those communications really needed to be
re-evaluated to make sure that they -- the tone was
appropriate for the time.
So our plans for the future really are to expand all of
this work.
So we need to use -- make broader use of all of our
platforms.
I think we need to make better use of data and
analytics so that we can assess a given campaign.
And modify it to get a better long-term impact.
And I think we also need to do better targeting of
outreach to institutions and institutions not highly
funded by NIH.
So these are the principles that Noni mentioned that
CSR operates with.
Our research is always specific, intentional and done
with these principles in mind.
This third one in particular in terms of multi
directional communication, that is something that my
office takes to heart.
It's the core of our work.
It's in our responses via email.
It's in our responses on Twitter and responses to
people tweeting directly.
It's really the core of everything we do and it's what
makes CSR better and it helps make review better, as
well.
So before I end, I want to acknowledge those who helped
carry out this work. Within my office, I have two
associate directors, Chuck Dunn and Lamont Williams.
Ann GraHamlin-Hamlin .
Lynndy-Ann Thai supports all of on our efforts.
Software development, video production, they do many
things.
And then the CSR outreach committee comprises SROs who
have an interesting in outreach and spend a lot of time
doing outreach on our behalf.
That's chaired by Jessica Smith.
And, of course, Dr. Gabriel Faso who helps connect with
diverse audiences.
With that, I'll stop sharing my slides and I'd love to
have your input on areas I think are missing and ways
we can expand.
>> DR. BRUCE REED: Thanks very much, Kristin.
It occurred to me too late that I should have given
some context for your talk.
>> DR. KRISTIN KRAMER: I'm sorry.
>> DR. BRUCE REED: That's okay.
I'll do it now.
For people who are new to council that don't appreciate
how much of a change this is for CSR, this has been an
area where we've invested a lot of effort and resources
in the last couple of years.
So one of the -- you know, so yes, we like to -- we
wanted to introduce you to the communications office,
but as I say, there is a direction that is very
important to CSR and we wanted to get some of your
feedback on the efforts of that office and where it's
headed, as Kristin said. Does it make sense where
we're headed?
Can we do better?
Those kind of things.
So thanks.
So Mark.
>> DR. MARK PEIFER: Mostly just a thank you.
I really think you guys have made a remarkable amount
of progress in this area.
And, you know, the idea of using all these different
ways of communicating with people I think is actually
critical, right? Because each person is seeing
information in different ways, right? So the blog
might be great for an old person like me, but Twitter
might be the place that a brand new investigator sees
things.
And just a kudo, because I think of all the of the
branches of NIH, you all are doing this -- you're way
at the time front of the line.
That includes any favorite institute, NIGMS.
They do okay, but it's mostly sort of boring.
Really boring.
I really think you guys are really pioneering something
that's critical. You are the face of NIH for most
people.
And even when I, who is on this council, right,
realizes I don't have really understand what is your
job and what is Mike Lauer's job, that's way true for
most people.
So I think you guys are doing a great job.
>> DR. BRUCE REED: Thank you, Mark.
Tonya.
>> DR. TONYA PALERMO: I completely echo Mark's
comments.
It's amazing the changes over the past several years.
I do a lot of teaching of grant writing and I pull up
the CSR website all the time.
Over the past ten years, just the evolution of that
website is truly amazing.
But I did wonder, I kind of feel like there's so much
good stuff tucked in there, how do you drive new
applicants there?
I haven't looked enough at -- the NIH website is
completely overwhelming.
But I don't know if there's strategic places to drive
people from there to CSR, especially around, again, all
of that, how to actually prepare an application
material that's so wonderful.
>> DR. KRISTIN KRAMER: So actually revising the
website and the information hierarchy on it is
something that I hope to accomplish in 2022.
Not so much thinking about applicants resources, but
more thinking about all of the data and evaluations
that CSR does.
Those things are really tucked away and is are very
visible.
So my goal is to pull that up to be a more top level
content item.
And in terms of applicant resources, I think the way he
we might expand would be doing -- we have plans to film
a new mock study section, a video that we can post.
But I think in a lot of cases, we have pointed to OERs
material on how to prepare grant.
So I think we're going to do more pointing to OER for
some of that material rather than duplicating their
efforts.
>> DR. BRUCE REED: Michelle.
>> DR. MICHELLE JANELSINS: Great presentation. I was
also kind of in awe about the efforts on communication.
And I think especially, you know, during the pandemic
when a lot of people have been feeling isolated, you
know, a little bit less so now, but, you know, it's
still, you know -- we're still going through tough
times.
And so I was actually thinking and it occurred to me
when you said the sentence that we read and respond to
every email or message. That could be your banner.
I think a lot of times, you know, people see blogs and
they think, oh, if I make a comment, no one is going to
listen to me.
But you're really digging deep and, like, reaching
people on their individual concerns.
So, you know, thanks for that and I do think that could
be, like, a possible banner or one of your rotating
banners. Because I was struck by that comment.
So thanks.
And I guess just for a question, you know, do you ask
for upcoming topics that people might, you know, want,
you know, to hear from CSR about -- or anything like
that?
>> DR. KRISTIN KRAMER: I think that's a good idea for
the future, but I'll be honest, we have a backlog of
blogs that we're drafting.
Right now we have a lot of content to get out.
But I appreciate your point about being responsive. I
feel like that is terribly important to do to respond
to people.
NIH is such a behemoth and it's difficult to navigate.
A lot of times, I just -- I want to make sure that we
respond to everyone.
And we might not be the right person to answer the
question.
We might be referring them to OER or some other group
at NIH, but it's really hard to figure out who to
contact.
So I really try to respond.
>> DR. BRUCE REED: Leo.
>> DR. LEOPOLDO CABASSA:.
I wanted to -- Michelle got my comment, the response is
fascinating to see and that I am so glad that your
office and your group is doing that and being very
thoughtful about that.
So I would highly recommend to continue to do that best
as you can.
The only other component I saw that was fascinated and
I'm glad that you're using data analytics here.
So where people are landing on the page and really
targeting information where people are most going.
Because I think that is a really important piece for
all these NIH websites which are enormous to navigate.
So I really appreciate that.
And excellent work on doing that.
Looking forward to see how this evolves and how you're
using those analytics to continue to improve.
So excellent job.
>> DR. BRUCE REED: Thanks, Leo.
Jinming.
>> DR. JINMING GAO: Kristin, I want to echo the
positive sentiments from all the council members.
And I really want to thank part of this process is
communicating what study section do SRO, you know,
responsibility to the outside community and really
appreciate the inside CSR so that we can see how
thoughtful you are in terms of thinking about the tones
and transparency and then explain the thinking process.
And I think it's easy for the council member to
appreciate, you know, how much thought and effort
you've gone through.
But at the same time, I think SROs, you know, those are
the counterparts of the reviewers inside of CSR.
How do we get the -- I guess, you know, words out so
that the outside reviewers, you know, the scientists
appreciates how much of the SROs actually are doing all
this behind the scene.
I think this is probably very important to -- I think,
you know, for many of us who want to pick up a phone
and complain, there's so much work behind the scene and
to appreciate all the efforts and the thought process I
think is good to let the outside world know what, you
know, you are doing.
>> DR. KRISTIN KRAMER: Yeah.
We recently had an article on the role of an SRO and
biospace.
I think there's a place to do more of that sort of
outreach to let people know what the role is of SROs
and to make sure that they're approachable.
We've tried to do some things in the past to humanize
CSRs such as with one of our revamps of our website.
We made photos of SROs a much larger and prominent
position on the study section description pages. That
was done with the goal of really humanizing, you know,
CSR so that is there is a person you can contact and
it's clear.
But I think we have more work to do in that space.
>> DR. JINMING GAO: Thank you.
>> DR. BRUCE REED: Thanks.
Narasimhan.
>> DR. NARASIMHAN RAJARAM: I just wanted to echo what
Mark and the others have said.
Great job on a lot of this stuff that you presented.
I am fairly active on Twitter and I see -- it's always
nice to see CSR responding to people who have comments
or comments.
It humanizes the agency and let's people know that
someone is reading their tweets and they're just not
sending this off into thin air.
I really like the data driven approach where you
figured out that putting stuff on the -- directly on
the study section pages was more impactful than having
it on the -- I realized that I never have visited the
front page of CSR.
I do exactly what you said, study section, name, put
that into Google and go with that.
>> DR. KRISTIN KRAMER: I can see it in the analytics.
>> DR. NARASIMHAN RAJARAM: I think that the -- given
how seamless, I was going to say this after Noni
finished her talk, just given how seamless the
transition was from in-person to virtual during the
pandemic, I don't know if there is -- maybe there is
here, but I don't know if there is enough appreciation
for how monumental the task was and especially given
that in some other countries that the review process is
completely shut down.
>> DR. KRISTIN KRAMER: Yeah.
>> DR. NARASIMHAN RAJARAM: So I know Mark posted on
Twitter, but reiterating that fact, how many
applications were reviewed and the fact that we didn't
drop a single cycle.
Not in sort of beating our own trumpet, but mostly as a
fact.
>> DR. KRISTIN KRAMER: Yeah.
I agree.
>> DR. BRUCE REED: Thanks, Matt.
>> DR. MATTHEW CARPENTER: Wonderful presentation.
I applaud your interactivity.
So thank you very much.
My question is about how to assist investigators from
finding the right study section. You know, the example
has been given about Googling one specific thing.
I'm guilty of that, as well. When you do that, you
don't know what you're missing.
>> DR. KRISTIN KRAMER: Yeah.
>> DR. MATTHEW CARPENTER: I know that my research
posted one of two study sections. I don't know what
else is -- my options, one you're alphabetical, I'm
wondering if there is a way we can help key words, that
kind of thing.
>> DR. KRISTIN KRAMER: It's just a referral tool.
And that's the name for it.
It's not -- I think a very intuitive name.
We tried calls it the guided study section selector and
that didn't take off because our own staff already knew
it as ART.
But, yeah, on the landing page is a place where you can
search. You can put in your specific names and title
and abstract. It will tell you recommended study
sections based on the last three rounds of referral
data.
>> DR. MATTHEW CARPENTER: That's interesting that you
can enter your abstract.
That's fascinating.
I think I just found what you're talking about. In
fact, maybe it could be a little more clear. Because I
typed in one word not knowing I can type in an
abstract.
Well done.
>> DR. KRISTIN KRAMER: So we need to publicize that
more.
That's again where we can probably use Twitter better
in advance of grant deadlines, sending out more tweets
saying hey, if you're trying to figure out what you
should put in, your assignment request form, go to this
tool or we can provide guidance on how to use NIH
reporter and matchmaker.
That is another great way to identify study sections.
>> DR. BRUCE REED: All right.
I'm going to take Christine and then Lynn and then
we're going to close out and move on to the next topic.
Christine.
>> DR. CHRISTINE HENDON: Thank you again, Kristin, to
you and your wonderful team.
I had a question regarding outreach to investigators,
primarily undergraduate institutions such as liberal
arts colleges or women's colleges.
So could you comment on that?
>> DR. KRISTIN KRAMER: We haven't done a lot of that,
I'll say. Most of our targeted outreach to
universities has been looking at minority-serving
institutions or institutions that are basically R-15
eligible and reaching out to those.
So I think that the way we've been finding contacts
would have excluded primarily undergraduate
institutions because the way I've started out is by
looking at institutions that have had some interactions
with NIH, but not a lot of interaction with NIH.
>> DR. CHRISTINE HENDON: I see.
>> DR. KRISTIN KRAMER: So I'm not picking up a lot of
schools that haven't submitted fellowships or
applications or research grants.
So that's an area that we could potentially expand to.
>> DR. CHRISTINE HENDON: Thank you.
>> DR. BRUCE REED: And Lynn.
>> LYNN YEE: I can the shared interest and overlap
section is incredibly helpful. I had no idea about the
ART tool, either.
And so it's kind of cool to see this. And I wonder if
the shared interest in overlap page could be also sort
of advertised or -- I mean could be a place to link
that tool.
>> DR. KRISTIN KRAMER: That's a good idea, yep. We
can do that.
>> DR. BRUCE REED: All right.
Thanks, everyone.
Thanks, Kristin and thanks for the discussion,
everybody.
Okay.
So looking ahead at the agenda, the next two items are
reports of ENQUIRE reviews.
I'm going to have Ross Shonat present ENQUIRE cluster
14 and then we're going to take a break around 3:00 and
we'll come back and Ray Jacobson will present ENQUIRE
Cluster 10.
So just to make sure that everyone has a basic
understanding of what we're talking about when we say
ENQUIRE, let me show one slide and give you a very
brief background on what the ENQUIRE process is.
So ENQUIRE is an acronym.
And I couldn't even begin to give you its derivation
now.
But think of it as a label for the process that CSR
uses to address the basic challenge of keeping our
study sections aligned with the progress of science.
So as scientific ideas and methods evolve as topics get
more and less popular, we have to change our study
sections.
So that they're manageable and so that they provide a
good review context for every application.
So the way we do this is that we take our hundred --
170 or standing study sections and we cluster them.
We put them into clusters of scientifically related
study sections.
Roughly about a dozen in each, but can go bigger than
that.
We put them into clusters.
We run each of these clusters through the ENQUIRE
process.
This is a fairly time and resource-intensive process.
We do about four of them a year.
And over the course of five years, that means we will
get through all of our study sections and then we will
go back and do it again.
So the process works in this way.
It starts by -- it starts with -- come on.
There we go.
So we submit the cluster to an external scientific
working group.
These are -- this is a relatively small group of
scientifically broad scientists.
They're not representatives of individual study
sections.
They're more broad thinkers in the field.
We asked them to consider the scientific that is
reviewed in that cluster of study sections and to give
us the best set of study sections that should cover
that area of science.
So they work out a list of study sections and
preliminary descriptions.
We give that to on internal trans-NIH working group.
These are senior-level IC and OD stakeholders.
It's also senior CSR leadership.
And we ask this group to basically start with the
recommendations of the external group and see if they
concur or if they have concerns about how this would
play out in review.
Based on the input of those two groups, we make a
preliminary list of study sections that should handle
that cluster.
And that is where we are today.
We bring this to CSR council.
We ask you to look at it and we want your high level
review of these.
Assuming that we get approval, we then take that set of
study sections forward.
We do some more intense work on them.
We do mock-sorts to look at the numbers very carefully
and we refine the descriptions of the study sections.
Eventually heading for implementation about six months
down the road from there.
So that is where we are today.
And so with that, I am going to turn this over to Ross,
who is going to give you -- who is going to talk to you
about Cluster 14.
So Ross.
>> DR. ROSS SHONAT: Thank you, Bruce.
So if you can see my slides, I'm going to be presenting
Cluster 14, a series of study sections addressing
microbiology and infectious diseases For background,
there are ten study sections in this cluster.
Four of focused on a bacterial pathogens, two in
virology, two largely in the area of tropical medicine
and some pathogen agnostic study sections related to
post events and vaccinations on microbial diseases.
You'll see in parentheses, approximately the number of
applications that that study section has been seeing
over the last three or four rounds.
So this is the external workgroup members.
It was chaired by Avery August.
Michelle was our CSR observer and I'm sure would be
happy to provide some context about what happened
during this meeting.
So here are the study section working group was --
looked at these ten study sections and began by
thinking about some of the overarching
considerations that they wanted to address.
I'll list a few of them here.
They largely were asked the question about how study
sections should be organized.
And so a lot of them are focused narrowly on a
particular pathogen.
But science has become much more multidisciplinary.
Some sections are become overcrowded, particularly
those focused on viral research.
They felt the need to better focus tropical, and noted
there were no obvious places to investigate microbe or
poly microbe interactions.
With that, I want to go through their recommendations
by group.
And I'll start with what they thought about the
bacterial pathogen group.
So as you see, there were four study sections in this
group. PCMB, BACP.
The acronyms probably have little meaning, but what
they effectively did was reconfigured some of these,
move some of the guidelines around and I'll explain
that.
So they came up with prokaryotic biology, which was
looking at the bacterial pathogen from a pathogen
perspective and BV was middle ground, BHI was bacterial
host interactions.
And then they created a brand new study called IMII,
interspecies microbial interactions and infections.
I'll have more to say about that in a moment.
And you may have noticed that we were running a large
overflow, bacterial SEP, and those applications were
effectively dissolved and redistributed around these
new study sections.
In the area of -- in the viral pathogen group, we were
running two study sections. You can see they are
overloaded.
Our typical size for a study section is around 60 to
80. So the external working group effectively crafted
three study sections out of two.
Again, along this broad distribution across from
molecular all the way up to host and human level.
Same broad themes as that defined for bacteria.
I'm going to jump ahead a little bit and talk about the
host-focused vaccine groups. There was IHD and VMD.
Both of these are oversubscribed.
And so largely what they did, they liked these study
sections.
They liked how they worked, but they proposed some ways
to reduce their scope, moving some of the pathogen,
more pathogen-focused applications out and to keep it
more -- IHD more immune-focused.
And with VMD, thinking about where this line was
between vaccine developed and virus focused.
All right.
Now I would spend a little bit more time on what they
thought about tropical diseases.
There were two study sections in this cluster in this
section to start with.
One was called PHET.
That's a eukaryotic section.
Another was vector biology. From those, they created
three new study seconds.
One was called medically important fungi and medical
mycoses.
They chose to take the mycology applications out of
PTHE and created a section focused exclusively on
fungi.
And then they created PZVA and PZVB which stands for
parasitic, zoonotic and vector-born diseases.
They imagined two of those study sections.
The overarching theme for PZVA and B is an emphasis on
their complex live cycles including the environmental
drivers and One Health Concept.
So PTHE got split into MFMM and the others went into
PZVA and PZVB.
And then I also wanted to note that PZVA and B are not
just parasites, but they would include any pathogen,
viruses, bacteria where the emphasis is on this complex
lifecycle.
Since they spent a lot of time reimagining this
interspecies and complex lifecycle group, let me spend
a few seconds outlining what their interests were.
So these three study sections, IMII, PZVA and B are
not focused on a single pathogen, but investigate
interspecies and complex interactions. IMII handles
things like biofilms, microbiomes, those type of
situations. PZVA and back seat are complex live cycle
focused with A focused at the genome and cellular
levels and B focused on community and population
levels.
So here is what the group came up with in the end.
They took the ten applications that you see in yellow
on the left and they imagined 12 study sections on the
right.
And I wanted to just kind of quickly go through more
visually kind of what their recommendations were.
So for the bacterial group, you'll see largely a lot of
the bacteria remained in this grouping of three study
sections. They would -- the SEP was divided across the
three study sections.
And then there would be some that would be peeled off
into these interspecies and complex lifecycle group if
that was an appropriate driver of that application.
In the viral group, they did more or less the same
thing.
2 became 3 plus pulling out the ones that are more
complex interspecies.
On the host focus, what I mentioned was the idea was to
reduce scope slightly.
And they did this by imagining and thinking about the
way we could redefine the guidelines such that it was
more pathogen focused, that it would go back to one of
these pathogen focused study sections and that would
bring those sizes down.
And then I want to just touch a little bit more on kind
of the big picture stuff for the eukaryotic vector
biology group.
So they proposed the TPHE would more or less be split
into two, the fungi and then PZVA and PZVB.
And they imagined and spent a great deal of time
discussing the concept that if you did this, you would
grow these study sections.
So I have indicated that here, that they would imagine
a lot of applications coming from outside the cluster,
new applications, people would be wanting these types
of environments and so that would help grow those two
study sections.
So the internal panel -- this was stretch two in
Bruce's outline.
They met in February and these were largely program
staff and directors at mostly NIAAD and NMS which is
largely where they study sections reside.
Let me go through quickly what they agreed with us with
the external panel.
So they were supportive of the recommendations to
restructure bacterial viral study sections. They had
some minor suggestions to clarify scope and overlaps.
They were supportive of recommendations changes to IHD
and VMD.
They thought we should rename them because we're
changing the scope slightly.
That makes perfect sense.
They welcomed the creation of a microbial-microbial
interactions study section, and finally with regard to
PZVA and PZVB, felt chat including these environmental
and population drivers would be welcomed.
So they also had some concerns, and I'll add the three
large ones here.
One was the mycology study section MFMM would not have
enough applications and that the science would be too
narrow.
They were concerned that all the parasites previously
housed in the PTHE would be reviewed alongside the big
ones, the viruses and vectors and maybe disadvantaged
in the new broader study sections. They were concerned
that pair parasite biology would no longer have a
review home.
As an alternative, they thought maybe if we were going
to move forward with PZVA and PZVB, that PZVA could be
the parasitic focus while PZVB would have a
bacterial/viral focus.
We spent a little bit of time doing some analysis to
see could we create, would there be enough applications
to create an MFMM study section.
What you see is the distribution of applications in the
fungal area based on where they are assigned to the IC.
As you can see imagine, NIID received the vast majority
of them.
But study section, you can see PTHE received most of
the fungal applications, but they are distributed
across a number of study sessions. ODCS, for example,
has a dental aspect and includes oral conditions that
might include fungi.
However, when we added up all the numbers, we currently
do not believe that there are enough fungal
applications across all CSR study sections to support
the creation of MFMM.
So I come to you with two possible scenarios, and we
can discuss as many scenarios that you think might be
available here.
But scenario 1 would be to go with what the external
panel recommended. They had some creative ideas.
They really rallied around these ideas and thought it
would be wonderful for the communities, particularly in
the area of tropical medicine.
Proposal number one would be to create MFMM as proposed
with the expectation of growth, with the study section
grow to be large enough down the road.
Scenario number two I offer up as a possibility would
be that we would leave PTHE alone, that is fungal and
parasitic applications would remain in the study
section. MFMM would not be created.
The P in PZVA and PZVB would disappear and that we
would need to do some additional analyses and mock
sorts to determine whether we would have enough
applications to support two study sections given that
the parasitic biology would be back in PTHE.
With that, I will pause for any discussion, comments,
feedback
>> DR. BRUCE REED: Thanks, Ross.
So reactions from council?
This is an area where we would, at the end of the
discussion, like a council action.
Michelle.
>> DR. MICHELLE JANELSINS: So I'll make a couple of
comments as an observer. I was able to observe the
meeting and I thought that, you know, it was very
engaging group, definitely the expertise needed was
there at the table.
I think that the group did really consider the size of
the -- like the applications within the study section a
lot and they discussed that a lot and how to
reorganize.
Then also I thought the group was very forward thinking
about, you know, where is the science going?
Where is the future going? There are definitely, you
know, interesting scenarios that could be happening I
think in tropical medicine.
Considering dynamic and interspecies dynamics and
really across all the different pathogens.
So I thought they did a really good job coming up with
these proposed study sections.
>> DR. BRUCE REED: Thanks, Michelle.
Mark.
>> DR. MARK PEIFER: I'm coming from well outside this
area. Just a couple of thoughts, right?
One is in general, I love what they did.
Really sort of thinking more big picture, not so
organism focused, but really question focused.
And I think that is present across almost all of the
changes that they initially proposed.
The one striking counterexample is trying to put all
the fungal applications in one study section.
So that seems counter to the overall broadening of the
biology.
The second thing -- and, again, you guys are probably
thinking about this, but I think it's important that we
realize that diseases that might be at the top of our
mind in the U.S. aren't necessarily the same diseases
that are the most important worldwide.
Again, this whole area of, you know, parasites and
probably fungal infections, too, things that have a
major, major impact in other parts of the world and not
so much of an impact in the U.S., I do hope we keep
homes for them, right? Because that is really
important work.
>> DR. ROSS SHONAT: Yeah.
So you noticed in one of my first slides, I said the
group tried to think about the organization of science
that wasn't pathogen focused.
But these pathogens are very different.
And you'll notice we do have pathogen-focused study
sections, the bacteria and the viruses happen to be
large components and they live within two or three or
four different study sections.
The interesting thing here is the two pathogens that
can't sustain themselves as the stand-alone study
sections are the parasites and the fungi.
So I think if there were enough applications, I think
the -- that the existence of a mycoses study section
would be different than what is being done with the
bacteria and the viruses.
>> DR. BRUCE REED: Sean.
>> Dr. Sean Davidson: I'm impressed with the rigger you
have put into this.
I think science is constantly changing and requires a
process like this.
So kudos to you guys, CSR in terms of whoever came up
with this idea and the people that are implementing it.
As a more specific comment, I have a fungal pathologist
who has an office just a couple doors down from me.
I've been hearing him for years talk about the conflict
between the fungal people and the parasitologists.
So I have a general knowledge of what this was going to
be based on the materials you sent me. And I sort of
explained to him that there is a thought about breaking
the fungal people off into separate sections with the
parasiteologists.
He likes that idea because he felt there was a
disconnect between how the fungal people think about
these oral nimps and how the parasitologists do.
And they don't have a good way of bridging across those
different organisms.
So I asked him if he thought this was a big enough
growth area to support his own study section down the
line. And he felt that there was because there is an
increasing issues with increasing immune result issues
that can result with these fungal problems.
That's an N equals one, but he said if that change was
implemented, it would be very popular, he thinks,
between the parasitologists and the fungal people.
I thought I would throw that out there.
>> DR. ROSS SHONAT: That resinated with the external
panel, as well.
They recognize the conflict among those two
communities.
And I think what we're ultimately -- we're receptive to
the idea, and I think that is what resulted in the
creation of this study section.
For us, though, we need to have study sections
appropriately sized with good competition, etcetera.
We don't have that so it largely becomes a management
issue for us.
What was pointed out to me, those organisms live
in different kingdoms and the only thick they share is
a nucleus. So it becomes then the SRO has to engage
fully with these two communities to make sure that
they're playing well together.
That is something that we can probably continue to do
and work harder at until perhaps the day comes when
there are enough fungal applications for it to become
its own healthy standing study section.
>> DR. BRUCE REED: Karen.
>> Karen Anderson: Thank you. This was well informed.
I have two comments.
One is when it comes to this, the recognition of the
ecology that influences the disease development and not
just a single pathogen, but certainly the microbial
microenvironment, for example, in which it lives and
the evolution of that process is really central to a
lot of that pathogenesis.
And one of the challenges when we break out to disease
focused approaches or organism-focused approaches is
that that ecology, that genome gets lost within the
context of how that -- of how they develop.
And just as a thought as the field shifts.
And my second thought on this is all microbial
pathogenesis happens in the context of immune systems,
functional or not, diverse or not.
And that that intersection between immunity and
pathogenesis and how that fits, where does
immunologist perhaps studies across multiple organism
types fit within the context of perhaps of these study
sections.
And these are comments not requiring answers, but just
as thoughts.
>> DR. ROSS SHONAT: What you said resinated strongly
with the group in that a lot of their innovative ideas
here were related to this interspecies /complex live
cycle interactions and they felt strongly this is where
the future was going, that this was how they could
create study sections that weren't so pathogen specific
and perhaps too insular.
>> DR. BRUCE REED: Jinming.
>> DR. JINMING GAO: I'm not an expert in infectious
disease.
When I was the chair of the gene and drug delivery
section, so we do have a diverse group of viral people
for doing gene delivery and then the polymer synthetic
folks doing a non viral gene delivery.
I think since the information is productive.
Where maybe the mNRA vaccine is the merging of the two
worlds in terms of complexity and the practical impact.
I wonder this sort of a discussion on the parasitology
and then the fungi, without knowing the field, whether
that could be beneficial for having a joint, you know,
study section to cross pollinate some of that
information.
And Sean's comment seems to support that.
And then the other thought I have is having this sort
of a broad umbrella interspecies and a complex
lifecycle group seems like it's probably a good buffer
idea to invite, you know, across disciplinary different
applications, maybe let it play out so that we can have
the A and a B for further division if needed.
But I am more for maybe using option number two from
the possible scenarios.
>> DR. ROSS SHONAT: Yeah.
So I think I already mentioned, so when the fungal
group -- obviously, the fungal's applicants would like
to have a fungal study section.
That led to the parasitologists saying we would like
to have our own study section.
So you have to balance tout needs of these communities
to what is the best for the peer review process.
In terms of PZA or PZVB, or whatever those ultimately
become, we have to do additional analysis.
But the external panel did come one some clear
distinguishing features between those two, one at a
cellular/molecular level and one at the population
level and it's something called the One Health
Concept, which it has to do with the relationship
between humans, animals and the environment.
They saw great growth there for NIH.
>> DR. BRUCE REED: A quick comment and I'll get to you.
thinking about growth areas, CSR has not had much luck
with the if you build it, they would come concept of
study sections. That has just -- let's say we can
think of no examples where that has proved to be true.
Another way of saying this is that current numbers have
proved to be a better guide to some study section
viability than sort of future estimates.
Obviously, we adjust things all the time based on what
happens, but it's proven very difficult to make these
sort of guesses about, you know, applications
increasing if you structure a study section a certain
way.
Edda.
>> Dr. SPIEKERKOETTER: What is really limiting the
number of applications from the fungal group? Is it
that they are in this one study section with the
parasitologists or is there a different reason why
they're not so much applications?
And then do you have any data on whether there are less
applications being funded because they are in this
joint group because then the -- maybe there are more
reviewers from the parasitology group that disfavor
the other applications?
Do we know any of those data that would require maybe a
separate group to protect the fungal, you know,
researchers, something like this?
>> DR. ROSS SHONAT: Yeah, yeah.
So I think we do need to do some more analysis of where
the applications are coming from, where they would be
coming from if they stood alone.
So that was one of the arguments that was made by the
external panel was that the mycologists need their own
study sections and if you build it, they will come.
We don't know whether that is true. Are the
parasitologists overwhelming the fungal community?
There are some arguments that that was true.
What is the solution to that?
As I mentioned, I think we need to make sure that those
two communities are working together to do review
fairly and are not siloing their reviews.
We have lots of study sections where -- not a lot. We
have a number of study sections where two communities
that don't get along are put together simply based on
size and need.
And which then becomes a management challenge for us to
ensure that we recruit appropriately across the sign
appropriately and manage expectations.
So I don't know where the answer is, but over the
course of the last six months that I've been involved
in this particular inquiry, the fungal and parasitic
divide has become very clear to me.
>> DR. NONI BYRNES: Let me jump in really briefly,
Ross.
I think we're past time here and we do need a vote on
options. If you decided to vote on option two, which
is not forming the mycology study section, you have to
rest assured that we will take a very careful
management look at the current existing study section,
which has both parasitology and mycology. We have
the data. It is not difficult to do the data scoring
analysis. We will have that in hand and we will
proceed with that in mind to make sure that no group is
being unfairly disadvantaged.
Historical issues aside, moving forward.
>> DR. BRUCE REED: So my sense of this discussion is
that there is a lot of enthusiasm for the general
framework and most of the study sections that the --
that were put forth by the external group.
The question comes down to -- Ross put the options on,
option one, option two.
I've heard a couple of comments from council sort of in
the favor of option two. Is that a sense of the
council?
Two, yes, I'm seeing that.
Would -- could somebody put a motion forth, then?
Michelle.
>> DR. JINMING GAO: I put a motion for option two.
>> DR. BRUCE REED: Okay.
Jinming.
And Michelle, you had a comment?
>> DR. MICHELLE JANELSINS: I had a question about if
we went with option two and if it came with the
opportunity for a fungal SEP to be formed.
Like if there became an abundance of applications,
would that be considered or would that be one of the
management approaches ?
I'm just wondering if someone can respond to that.
>> DR. NONI BYRNES: Let me answer that right away.
If suddenly there is a big growth of applications in
one area, we're trying to approach our overall panel
sizes to have better review outcomes and better review
management. We will create SEPs as necessary, yes.
>> DR. BRUCE REED: All right.
So the motion is to accept the recommendations for the
external panel with option 2, with the addition of
option 2.
I think we know what we're talking about here.
So let's just have a vote. Could I see a show of hands
of people -- council members in favor of that.
All right.
Do I see any opposed?
Not seeing any opposed.
Michelle, is your -- is that a vote or --
>> DR. MICHELLE JANELSINS: Yes, I raised my hand.
Yes, I agree with option two.
>> DR. BRUCE REED: You agree with option two. Okay.
Great. Thank you.
So well, we're a little over time, but we all need a
break.
So we're going to take a break.
We'll take ten minutes.
It's 3:10 by my clock right now. Let's start promptly
at 3:20.
All right?
So I'll see you back shortly.
Next up on the agenda is the report from ENQUIRE
Cluster 10. This will be presented by Ray Jacobson.
Ray, you're up.
Thanks, Bruce.
I'm going to take the next 15 minutes and hopefully
only the next 15 minutes to provide the council with an
update about a second ENQUIRE process that's been
ongoing at CSR since late last year.
This is one I've been managing that has to do with the
drug discovery process, which is ENQUIRE Cluster 10.
It's a little broader than just drug discovery.
It involves drug and small molecule synthesis,
environmental toxins and xenobiotics, as well.
So I'm going to start off by showing you the study
sections involved in Cluster 10.
So this slide shows the set of the panels that were
included.
There's 11 chartered panels here.
And 2 recurring steps at the bottom of the slide.
BST(55) and AIDC(82).
So topics range from basic science to synthetic
chemistry All the way to preclinical pre-IND studies
for oncology therapeutics.
So when you look at this slide, you can see that the
study sections range in size from very small. BST(55)
has 39 applications. XNDA is at 56.
And some of them are quite large.
So DDR, drug delivery and mechanisms of antimicrobial
resistance is 81.
This is historical data for the applications of these
panels over the last few years, last three years.
So there's been about 250 applications or so coming in.
So they've been split off to keep the panel size
manageable for DDR.
So if you total up the numbers of applications that are
coming in, there's roughly less than -- about 1100,
slightly less than 1100 applications.
For an ideal study section size, that would be enough
for 14 to 15 panels and at the moment we have 11
chartered panels and a couple sets of SEPs.
So I want to tell you a little bit.
This is probably familiar to most of you to talk about
what we asked the workgroup to do.
We brought in 12 members who had experience as either
review or applicant across multiple study sections
within this group of panels.
The members were given representative sets of
applications titles and abstracts from each section,
study section guidelines and workload information over
time.
The external workgroup was charged with examining
existing study section organization and recommending
changes to optimize the study section size and
function.
And those changes that we were going to ask the
workgroup's recommendations for could include
modification of referral guidelines and boundaries,
adding emerging fields, creating new study sections,
disbanding study sections or merging and redistributing
topics.
So we assembled an external workgroup and they met
December 13th and 14th last year.
The workgroup members were drawn from a wide range of
institutions and had expertise spanning the topics that
were included in this. Kip guy, the dean of the school
of pharmacy at the University of Kentucky acts as the
chairperson and we were fortunate to have tone ya
Palermo serve as the observer from the advisory
council.
The individuals covers basic science from synthetic
chemistry all the way to drug delivery and advanced
therapeutics.
So the first thing we did was we asked them to give us
their thoughts based on the examination of the data we
sent them and their experiences from serving on these
panels or as having their applications reviewed there.
And we asked them to come up with their observations to
help drive the discussion in terms of what we should do
to put together new panels.
So I will list what the sort of high level ideas that
were identified at the beginning of the meeting.
The panel noted that there's often a tension between
the basic science applications and more applied in
translational topics, both in the chemistry panels and
the drug discovery panels.
This is a common feature in a lot of our study
sections.
But they noted that was prevalent here.
Very early screens to identify potential drug leads
would be better served in the chemistry panels rather
than in the later stage drug discovery channels, DDR or
DMP.
At the present time, if you were doing drug discovery
in a panel in a different biological system, there
isn't a place for those applications to go, even though
the methodologies they use for drug discovery are all
similar.
And then the panel also noted that an aspect of the
drug discovery and resistance study section currently
looks at anti-infected resistance and that's an
important aspect of drug efficacy that needed to be
considered when they looked at the new study section in
the space.
In terms of later stage drug discovery types of
applications and delivery, there was a feeling that
there needed to be more support for late-stage drug
development pre-IND work, that there wasn't enough
capacity there.
They also mentioned that biologics, not nucleic
acids, perhaps monocolonal antibodies, it was pointed
out that biologics are a growing area, but they
often are forgotten about There was an expectation that
there would be growth in the area of mRNA based
therapeutics and this was likely to be a place where we
might see more applications come in the future.
And then the last element in this area was that there
also isn't a place for advanced therapeutic studies
that don't involve cancer.
So there is a developmental therapeutics panel that
looks at later stage drug development types of studies,
but that only reviews applications related to
cancer-based topics.
And then there was one additional comment in the area
of environmental toxins that the workgroup noted that
currently environmental neurotoxins are reviewed in a
neurotoxicity and alcohol study section, but there's a
tension in that panel and they felt it would be worth
us having a look at some point in terms of whether the
environmental toxins might be better reviewed in the
context of other environmental non neuro-based systems.
So the panel was asked to identify what the main ideas
were in the applications we sent them.
So on a whiteboard, we put up what they felt to be the
main topics.
They came up with 21 broad different topic areas.
Over the couple-day period, we could assesed those into
areas with what they were proposing as new study
sections.
They clustered these 14 new proposed panels into
conceptual framework involving a drug discovery
pipeline with three phases: Enabling discovery, drug
discovery, and drug development, which is the later
stage.
So in the next few slides, I'm going to go through very
quickly the different panels that they envisioned in
each one of these.
The council has the external workgroup's report that
has all the of the detailed guidelines for these study
sections so you can look at those in more detail at
your own pace.
So the enabling discovery phase is the basic science
part of the drug discovery pipeline that they proposed.
They suggested redistributing the topics that are
currently reviewed in the two chemistry panels so
create one panel that is more very basic chemistry that
involved synthetic and biosynthetic methods.
And a separate panel that would be directed towards
more applied type of chemistry, including chemical
biology and medicinal chemistry.
Those were called chemical synthesis and biosyntheses
.
They suggested redistributing the drug delivery and
nano science topics from the GDP and nano panels into
three.
Those panels are a little bit big at the moment.
And they pulled out the very basic nanotechnology
aspect material sciences and development of new nano
particles into a panel called innovations and non know
systems and nanotechnology that would look at the basic
science aspects of nano.
And they separated out delivery aspects that are going
to show up in one of the later phases.
And then they suggested expanding the topics covered by
the existing panel SIEE, which is systemic injury from
environmental exposure to more broadly study etiology
outcomes and mechanisms of environmentally induced
disease.
They called that panel environmental determinants of
disease.
The drug discovery phase, which they described as being
studies where early panics can be tested which are
currently reviewed in DDR, DMP and the recurring panel
SEP A, DC82, they suggested merging those applications
for those areas together to form four drug discovery
panels that each have very similar guidelines, but a
different biological system.
So DMPA, win focused on anti-bacterial, anti-fungal and
anti-parasitic drug discovery, DMPB which focused on
antiviral drug discovery, DMPC, which would be cancer
drug discovery and DMPD would be neuroscience and other
assorted biological systems that people were interested
in developing therapeutic compounds for.
And they also suggested creating a panel that would
focus discovery and drug efficacy area called
anti-infective resistance and targets.
The last thing they recommended was to absorb the small
BST(55) SEP which involves high throughput screening
applications into the relevant drug discovery panels,
DMPA through D.
The last phase was the drug development phase.
The remaining topics from existing drug delivery and
nanoscience panels, they included splitting those to
a panel that was focused on the nucleic acid
therapeutic delivery and a second that would focus on
small molecule and buy logic therapeutic delivery.
Topics currently reviewed in the XMDA panel, which is
basically limited to sympatic systems, they suggested
any cell type, organ system or anatomical
compartment, that would be called drug and buy logic
disposition and toxicology.
They suggested the broadly involving development of new
therapeutic strategies Forney row plastic diseases,
solid tumors and leukemias, were mapped to advancing
therapeutics A.
Therapeutics strategies would map to the therapeutics B
panel.
And then the last study section included in this grew,
cancer etiology which has mitogens that lead to
oncogenesis, they didn't see that as being released to
the drug discovery pipeline.
It's related to SINDE but it's not part of the pipeline
that they envisioned.
But the panel members recommended that the topics
currently reviewed in CE were basic cancer biology that
very well could be reviewed in the context of other
basic cancer biology study sections.
Those are currently being organized via ENQUIRE from
cluster 5, which you heard about at the last CSR
advisory council.
So the plan is to absorb the bullet points from the
cancer etiology study section.
Those topics will be added to the new sections
currently being formed.
And those will be implemented for the January 2023
council round.
We assembled a internal workgroup meeting that met
February and March.
The internal panels includes the CSR division
directors, IRG chiefs, senior level individuals from
the involves ICs listed there.
The internal panel was asked to look at the
recommendations from the external workgroup for
potential issues that would arise if we were to
implement them.
They looked at survey information from POs and
reviewers about the existing panels as well as notes
from observers who visited the sites, the study
sections.
The internal panel recommended, they expressed
concurrence with the general recommendations from the
external workgroup.
They really didn't have any issues that they raised.
There were a few minor modifications that they
suggestion.
They recommended instead of using the initial names,
which I didn't show on these slides, which were more
specific and included the biological systems in the
titles, that we replace those with general names, drug
discoveries and molecular pharmacology A through D and
advancing therapeutics A and B instead of biological
named panels for the reason it's very difficult to
include all the subdisciplines we might have going to
those study sections and these general names don't
exclude any particular community.
Then they said some minor adjustments including change
and review loci for some orphaned environmental studies
that were reviewed elsewhere and provided input about
referral overlap boundaries where the boundary is
between the synthetic chemistry panel and the drug
discovery panels.
In the end, there were 14 study sections proposed.
These would be implemented in May of 2023 if the
council agrees.
And these are listed here and I've gone through them,
so I won't read them again for you.
And then the last slide I want to show you would be the
proposed next steps assuming the council concurs. We
would refine if the draft guidelines and overlap
statements next month and through May. We would look
at mock-sock data using historical application data to
see what the study section sizes looked like and make
adjustments. We would post new guidelines to the CSR
website in August this year been which would give us a
couple of months before the first applications to
arrive to the first receipt dates where the study
sections would go live would be for the October and
November receipt dates.
Our current study section members would be transferred
to the new panels late this year, December of 2022.
And then the first meeting of these new study sections
would take place January and February of 2023.
And those applications that were reviewed during those
meetings would go to advisory council for May and June
of 2023.
Okay.
So I'm going to stop there and take questions or
comments and give Tonya a chance to give any input that
she would like to.
And we'll come back and I'll ask whether the council
concurs.
So I'm going to stop sharing my slides.
>> DR. BRUCE REED: All right.
So thanks, Ray.
Tonya, do you have comments?
>> DR. TONYA PALERMO: Quick comments as the observer.
This topic is pretty far outside my own area of
science, so definitely was in an observer only role.
As Ray mentioned, I thought this group did a neat job
of pulling in the conceptualization of how they wanted
to approach this.
Once they did that, they were able to fit in all of
these topic areas quite nicely.
The biggest job for the them was reducing the number of
topics that they had.
So, yeah, kudos to the group.
I thought they did an awesome job.
>> DR. BRUCE REED: Thanks, Tonya.
>> DR. RAY JACOBSON: I think Mark had his hand up.
>> DR. BRUCE REED: No.
Sean did.
>> DR. SEAN DAVIDSON: If you haven't done the mock
application sorts, how often do those change the
outcomes of what the internal people have decided?
And does it have to come back to this committee if
there's a big change or is that kind ooh a performer
type thing that gets done?
>> DR. RAY JACOBSON: The mock-sort data can affect the
outcome. Unless there was a large change to the
framework that was being proposed, we would not bring
it back to the committee.
So, for example, let's say one of the panels, DMPA
through D was too small because the numbers have
drifted since we looked at the historical data. We
would likely compress those through into three panels,
for example.
That has happened in the past and we have done that
because we ultimately need to have study sections that
are a functional size.
Karen.
>> DR. KAREN ANDERSON: Thank you very much.
It's obviously an impressive portfolio.
There is obviously a clear distinction between the
basic science and the development of the biology
development, cancer biology, microbiology and the
therapeutics development.
Is there any mechanism or effort to at least foster
collaborations between these or across thinking?
Because now if you're developing basic science of a
microbial pathogen and you're studying that, you might
not be thinking about the issues involved in
translation to therapeutics until five years after
you're completing your project.
And those can have, you know, clear delays in terms of
the development into therapeutics.
How do you balance that?
>> DR. RAY JACOBSON: That is a great point. And it
was brought up in the internal panel discussions, as
well, that you need some input of -- in items of input
from the translational perspective when your doing the
basic science, as well.
I think what saved things a little bit is that the
typically these people don't fall neatly into one box,
right?
So if you look at the chemists who observed, many of
the people who served, served on the basic science
chemistry panels, but they're pulled right now to
review in the sort of mid or later stage drug discovery
panels.
So people -- even though we have boxes where we put
things in terms of how we review things, I think
expertise of reviewers is often broader than our study
section guidelines.
It doesn't completely help and it's something that the
management perspective we have to keep an eye on.
And that is something in terms of how we look at
significance when we're asking to reviewers to do that.
That is something for the chair to make sure that input
is there, too.
>> DR. BRUCE REED: Let me jump in at this point. We
have a fairly busy agenda ahead and this is an area --
this was a review where the external panel came up with
a nice conceptual and very clear layout. And it seemed
like there was a high degree of concurrence from the
internal panel.
So I'd like to hear if there are any objections or
concerns from council regarding what the panels had
suggested.
And if we don't have concerns, then I'd like to move
ahead and get concurrence with those recommendations.
But I certainly do want to leave time if there are
concerns that anybody has got.
Jinming, you had your hand up.
>> DR. JINMING GAO: It is not a serious concern, but a
minor concern about the labelling.
The labelling discovery and the drug developments.
I would start from the drug development because I am
getting into helping translate the technology from the
lab into the clinic. I feel like then the typical drug
development, the definition, there are a lot of CMC,
you know, the chemistry manufacturing, a lot of the
practical end of the analytical thoughts development,
the QAQCs, I wonder whether that's a good name to
characterize.
To me, all of these different topics are still on drug
delivery technology, maybe aren't, you know, nucleic
acid delivery.
So it's not quite striking me as a drug development.
Maybe as an early early stage, but I just -- I don't
have a good name for it.
If I think of one, I can probably just email that.
And then the clarification on enabling discovery, I
wonder whether that is just focusing on all the effort
on the prediscovery side or is it more the technology
piece that, you know, instead of just enabling
discovery doesn't seem to come across as clear.
There are a lot of things enabling discovery, but I
wonder whether that is a -- the innovation and
technology part that's, you know, enabling the
discovery is a goal, but it's the focus is developing,
you know, innovative probes or, you know,
nanotechnology, nanosystems.
And then, you know, leading the discovery.
So also thinking that discovery might be better
phrased.
So two minor, I guess, suggestion webs not really major
concerns.
>> DR. RAY JACOBSON: So are you asking about the broad
category titles or the individual study section titles?
>> DR. JINMING GAO: Really the broad categorical --
>> DR. RAY JACOBSON: So no one will ever see those
again.
Those are basically a framework for us to think about
how to put them together. When you go on to the site,
it will be the individual study section names.
But your points are -- you know, it's hard to get a
name that captures everything at this point.
That's what we suffered from, probably.
>> DR. BRUCE REED: Thanks.
Okay. Can I get a motion from council on this?
I think this is -- I think I'm looking for a motion to
concur with the recommendations of the external panel.
External and internal, I guess.
>> DR. MARK PEIFER: So moved.
>> DR. BRUCE REED: Mark moved.
>> DR. NONI BYRNES: Second.
>> DR. BRUCE REED: Can I see anybody who is opposed.
.
Not seeing any, so that carries unanimously.
Thank you, everybody.
Okay.
Hope Cummings has been waiting patiently.
She has some interesting data to provide you.
Just for context, I think all of you know this summer
CSR pushed out training for reviewers on bias awareness
and bias mitigation and a peer reviewed context.
This is a very important effort to us. We are
interested in assessing how effective it is.
Part of that is getting feedback from the reviewers
themselves about their experience with this training.
Hope conducted a survey on this and she's here this
afternoon to talk to us about what she's learned.
>> DR. HOPE CUMMINGS: Thank you, Bruce.
The training was developed by a diverse working group
that included extramural sciences as well as staff and
CSR, the office of scientific workforce diversity and
the institute of general medical sciences. Several
members had background in expertise and bias and bias
training.
So to help reviewers develop ways to mitigate bias.
The training's heart is the most common bias in review
and was not explicitly designed to be implicit bias
training.
The overall focus is on bias related to the principal
investigator and institutional reputation and on the
importance of adhering to the review criteria in the
areas of investigator and environment.
The training is approximately 30 minutes long.
It begins with testimonials from real NIH grant
applicants who describe instances of bias that they
have experienced in review of their applications.
They move on and engages reviewers in a short activity
that illustrates the implicit biases that language can
elicit.
The next activity focusing on reviewing a statement and
critique.
Last is a study section that presents two instances of
a review of a flawed application.
The first demonstrates two versions in which viewers
intervene to mitigate the bias remarks.
So for the January 2022 council round, almost 10,000
reviewers who came from 412 study sections were asked
to take the training four weeks before their fall
meeting. They were sent a reminder email two weeks
before the meeting.
By the end of January, around 6,000 reviewers took the
training for a 60% response rate.
At the end of the training, reviewers were asked if
they wanted to complete a survey of the vacation
purchase we had a 53% response rate.
These reviewers represented 389 of the study sections.
An additional 5,300 reviewers have taken the training
for the May 22 council round.
The evaluation consisted of nine questions that measure
reviewer's general satisfaction with the training and
whether the training helped them become more
knowledgeable and aware of bias peer review.
If the training helped them feel more comfortable as
well as how they might intervene in the future and on
the usefulness of some of the specific training
activities that I described earlier.
There were a couple of other measures not directly
related to the training, but were included to help
gather reviewers' behaviors and perceptions regarding
peer review bias.
These measures included the presence of bias, their
ability to correctly identify bias and their frequently
intervening on bias.
The survey concluded with an open-ended field where
reviewers could share their overall comments to review
and recommendations for improving the training.
As I move to the results, I'm going to start with the
findings on reviewers' perceptions on bias and the year
prior to the training just to provide some background
context.
So if you look at the first graph, you can see the
great majority of reviewers believe NIH has a problem
with bias and review with 45% of reviewers believe bias
is a moderate problem and 42% think there's a big or
very big problem.
In the last year, they read a critique or attended a
review meeting where they felt bias was present.
55% of reviewers reported they sometimes did and 39%
said they never or rarely attend a critique or read a
review meeting where they felt bias was present.
In regard to reviewers ability to correctly identified
bias in the last year, about half were moderately
confident they could do so.
Approximately 30% were extremely confident in their
ability to identify bias in the last year.
The results show most reviewers, or 53%, never or
rarely intervened when they thought bias was present.
Only 4% of reviewers reported that they always enter
vined on bias in the last year.
Okay.
So now I'm turning to the results of the training
itself.
Overall, reviewers were highly satisfied with the
training. Around 90% agreed or strongly agreed that
the training met their needs and expectations, the
testimonials were believing and that the training was
engaging.
About assessing whether the training helped reviewers
many more knowledgeable, the results show around 60% of
reviewers thought the training helped them to a large
extent.
One, identify potential biases in review, two,
understand the importance of using the review criteria
to assess and avoid bias and review.
Three, write an unbiased critique or present an unbias
argument during a review meeting and last that the
training helped them become more aware of the personal
review biases that they hold.
Most reviewers, 60% say they helped to a large extent
across all of these items.
To follow, approximately 30% of reviewers report the
training helped them to a moderate extent.
This is across all four items.
So between these two categories, around 90% of
reviewers believe the trainings really increased their
knowledge around awareness and bias and review.
The results showed that reviewers felt the training
helped to feel more comfortable intervening on bias
with around 70% believe it helped them to a large
extent feel more comfortable.
Only a very small percentage of reviewers, 7% to be
exact, reported that the training had little or no
effect in helping them to feel more comfortable
intervening on bias.
Now beyond reviewers' feeling more comfortable with
intervening, the results showed reviewers were highly
likely to take actions in the future around bias.
Around 82% stated that they would definitely or
probably ask for clarification or justification from a
reviewer if they felt the reviewer made a bias
statement during the meeting.
Approximately 60% of reviewers would definitely or
probably contact the SRO or Chair if they perceived
bias in a critique or during a meeting.
In regard to the specific training activities, around
90% of reviewers thought the training activities
were very useful.
Quickly moving on to the qualitative results, here
reviewers were given the opportunity to share any
general comments about bias in NIH review or
recommendations for improving the training.
Given this it's an open-ended question, it tends to
require more thought and effort. 46% of reviewers
responded and provided around 1,500 comments.
The qualitative analysis identified 8 common and
salient themes present in interviewers's comments and
I'll be brief as I go through these. You can see more
details in the ran report. so the first theme
general satisfaction and value of the training.
So here the great majority of reviewer comments were
those of simple thanks and appreciation for the
training.
Reviewers applauding NIH for tackling bias in review
and were pleased with the high quality of training.
They stated they learned a lot from the training and
believed they could have a real impact in reducing bias
and review.
They felt that the training activities were relating
and were very impressed by the study section scenario.
Reviewers overwhelmingly repressed more scenarios that
they could watch and learn from and many thought it
should be required every year or in every review
meeting.
Many reviewers were motivated and empowered by the
training.
They stated the training helped them become more
confident in speaking up and out against bias.
The training helped reviewers recognize their own
biases and many made vows to correct their bias
attitudes for future reviews.
Requests for more direct approaches to tackle bias.
Reviewers found the bias mitigation strategies and the
training extremely valuable and they wanted more
resources and examples on how to intervene in
situations where different types of biases are present,
such as scientific bias and when bias is present in
different context such as with non discussed
applications.
Reviewers wanted resources and strategies to help
themselves themselves from expressing bias in review.
They suggested examples such as clarifying questions to
ask when challenging statements, different avenues to
raise bias concerns during the meeting, templates to
write unbias critiques, and a cheat sheet of common
reviewed language that was bias.
Reviewers noted the challenges and potential
consequences that many reviewers faced that could deter
them from intervening bias, primarily the power
dynamics associated with reviewers social and
demographic status.
Reviewers felt that young, less experienced reviewers
and reviewers from marginalized and minority groups
would be less likely to speak out against bias out of
fear of being ignored or undermined or because of the
potential for negative repercussions as an applicant
reviewer.
Reviewers recommended the training include more
acknowledgement and intervention strategies that took
these power dynamics into account.
If the need for chairs and SROs to intervene more and
receive more bias training.
Reviewers believe SROs and chairs hold the most power
in the room and setting the tone for acceptable and
unacceptable behavior.
As such was they believe SROs should take the lead and
changing review culture to one that does not tolerate
bias.
Many reviewers noted that they would feel more
comfortable and confident intervening if they saw the
chair and SRO play a more active role in intervening.
They thought the chairs and SROs could benefit from
additional training on bias given their unique
position.
Experiences with bias as a reviewer and applicant.
Reviewers stated that they could relate to the scenario
and the bias training with many stating that they have
been in meetings where these exact scenarios have
occurred. They also shared other accounts of bias that
they experienced or observed as a reviewer.
Reviewers said that the training highlighted many of
the biases that they have faced as grant applicants
mainly from reviewer comments on their applications.
Reviewer recommendations to reduce bias in review.
Reviewers gave their own recommendations on how NIH can
reduce bias on review.
Many looked at incorporating blind reviews and adopting
a stage review process.
Reviewers noted that increasing the diversity on panels
could help reduce bias.
Many reviewers suggested removing the environment and
investigative criteria since they were heavily
susceptible to bias.
Last, recommendations to improve training.
Aside from the overwhelming request to add more
scenarios, reviewers stated that the training could be
improved by covering more types of bias and by
addressing review biases that are more implicit in
nature.
Many reviewers wanted to learn more about scientific
bias and bias associated with the socio demographic
characteristics of reviewers and applicants.
Bias associated with critique score mismatch and non
discuss applications were of concern.
Finally, reviewers noted that the biases covered in the
training were explicit in nature and easily
identifiable and more likely to be addressed and
corrected. They wanted the training to cover more
implicit or acceptable form of biases that arise during
review.
So in short, reviewers applauded NIH for tackling bias
in review. They were highly satisfied with the
training and that it was very useful.
The training was very successful in increasing
reviewers' knowledge and awareness of bias in peer
review and it significantly helped reviewers feel more
comfortable in intervening on bias and it empowered
reviewers to actively take action in the future.
So our next steps include taking measures to increase
reviewer participation in the training.
Identifying opportunities to keep reviewers engaged
about bias mitigation and to continue to help chairs is
SROs play a more active role in bias intervention.
And last, for the next module, we need to determine the
next type of bias to cover and how to address biases in
more implicit ways.
Any questions?
>> DR. BRUCE REED: Thanks very much, hope.
Maybe you want to take down your slides and we'll open
it up to discussion.
>> DR. HOPE CUMMINGS: Sure.
>> DR. BRUCE REED: Let's see.
Tonya.
>> DR. TONYA PALERMO: Thanks so much, Hope. That was
a wonderful overview.
I just wanted to share I have heard wonderful comments
from my own colleagues who have gone through the
training.
I just wanted to know if there's thoughts now about
making this mandatory for reviewers and thoughts about
how that might look?
>> DR. BRUCE REED: Yeah. Noni, do you want to --
>> DR. NONI BYRNES: Yeah.
Let me take that.
Yes, I think that ultimately this has to become
mandatory. This was our first attempt out of the door.
We went sure how it would be received.
It looks like it's well received. We have places to
adjust.
And yes, that is planned for the future.
Not next round, but soon.
>> DR. BRUCE REED: Sean.
>> DR. SEAN DAVIDSON: I want to reiterate that I think
this is a great training program.
And I also reiterate that I think this needs to be made
mandatory.
It's an enjoyable training.
I think people, once they realize it's going on be well
taken to for everybody and there has to be a mechanism
to get people to do that.
The other thing I would suggest you do is think about
something like this for the spiel that the SROs give to
all their section members about how to be a good study
section reviewer and things like that with this
experiential type of a paradigm which I would hope would
help homogenize the experience.
>> DR. NONI BYRNES: Let me respond to that.
Our office of training has a number of innovative ideas
that are already sort of halfway in progress.
And I'm hoping that if not in September, then hopefully
by next March she would be able to give you an update
on what wear doing to make consistent the training for
SROs and reviewers, including updating this training
for the next iteration which I'm hoping we can make
mandatory.
Frankly, I think if we're going to make something
mandatory, it's simpler to go through the ERA systems
and have it be something people have to do before their
submit their application.
That is manage we need to the with OER.
>> DR. LEOPOLDO CABASSA: This is very helpful.
I think it highlights the work that has been done for
this and the follow through of understanding how this
works, what's working and what is not.
I definitely think that making this mandatory would be
helpful, but continue to create materials in small
handouts of the -- not only of the bias, but of the
activities and the actions that reviewers, SROs and the
chairs can do.
When you're going to review, we get all these handouts
on the scoring system and all of that stuff.
This should be part of that package, as well.
So it's continuously being reminded and highlighted.
And then I think the other component here, as we are
expanding the diversity of our review committees and
bringing younger investigators or early career
investigators, this is an important area of training.
That we -- not only diversity, but we need to empower
the younger investigators and the underrepresented in
order to what are the actions to take to be able to
address this bias.
Here are some examples and training around that.
So follow through on that will be really important.
Lastly, I think figuring out what the metrics are for
evaluating the impact that this is having.
So, for example, are there more reports or bias right
now coming after this training?
What other metrics you can think of of figuring out
whether this is working to make adjustments as this is
rolled out.
I commend you for doing this and I think it's going in
the right direction agency we're moving forward.
Thanks.
>> DR. BRUCE REED: Thanks, Leo.
Karen.
>> DR. KAREN ANDERSON: Thank you so much.
This training is terrific.
I think the concepts of the training is not a single
time point, but it is an ongoing, continuous education
process on behalf of all scientists and it is important
that the training be not just for viewers but for grant
recipients as they are the mentors of the next
generation that is going to be doing this.
And I think the opportunities here are terrific.
And my second thought is to echo the comment about
environment in particular, being an at-risk area for
overt implicit bias and thinking through those
criterion and how that might be reimagined sometime in
the future. Thank you.
>> DR. BRUCE REED: Thanks, Karen.
Lynn.
>> DR. LYNN YEE: I want to express gratitude for this
training program.
I think it's outstanding and it's great to hear that
others did, too.
I think that one area that could be a focus of future
trainings is as others have echoed already is a little
bit about the culture of power inside this study
section and who -- whether, for example, the junior
members of the study section, whether ad hoc or
standing members defer to people who have been standing
members for longer or are people they respect in their
fields.
And whether that deference and power, which is probably
true in many fields, but especially in my field, for
example, a surgical subspeciality are hierarchies are
intrinsic to the way we practice.
I think that power dynamic can influence the direction
of study section. And it would be interesting to call
that outline in future training else and not just try
to convince people to speak up, but to try to
acknowledge that power differentials exist and that we
need to work to overcome them.
>> DR. BRUCE REED: Thanks.
>> DR. NONI BYRNES: That is a really good point. I
see Hope taking notes.
I will say we are trying to incorporate this training
and more discussions around the power dynamic in our
chair orientations that we do once a year.
And I think the chair plays a critical role in
ensuring, you know, sort of mitigating that power
dynamic.
>> DR. BRUCE REED: Yeah.
And I will just say briefly add on to that, I'm not
sure it's always appreciated that we didn't simply roll
this out to reviewers, but we made a lot of effort to
help SROs get more comfortable with the concepts of
bias, to work with this training, to work with this
training with their reviewers because as many of the
peoples commented, the SROs play a critical role in
this, both in training reviewers and in the meeting
itself.
So we are paying attention to that aspect, as well.
Okay. Let's move on. Thank you, Hope.
We'll move on to the last section of the agenda, which
I think will be a lively discussion.
I hope it will be.
This is the report from the working group on review of
fellowship applications.
And let me just bring up my screen.
Okay.
All right. Hopefully now you see my screen.
Elizabeth and I are going to divide this presentations.
This report is an interim report.
And the working group is looking forward to feedback
from council about the ideas that they have developed
to this point.
So we are dividing this.
I'm going to go over some of the background that sort
of informs the group's thinking and Elizabeth will give
you some of their thoughts about how things might be
changed.
So just in terms of background, I think, I hope, one of
the themes of the meeting today is that CSR is
committed to improving the peer review process.
That is goal 4 of our strategic plan if
anybody is paying attention to that. This fellowship
effort extends beyond that. We have heard persistent
concerns that fellowship reviews may disadvantage some
highly qualified applicants.
These concerns led CSR late last summer to convene a
working group on fellowship review with the charge to
evaluate the fellowship review process and to make
recommendations to make that process as fair and
effective as possible for all.
This is the group.
It consists of some council members in case of Scott
Miller former council member, members of the NIH staff,
both from within CSR and across NIH, and then external
scientists.
The external scientists come from a variety of
institutions and represent different career stages.
They all had personal experience with writing
fellowship applications and with the review of
fellowship applications.
They have been a very active engaged group.
What I want to cover here is some of the external
input, some of the information that they considered in
thinking about where the issues are and how things
might be changed.
So one of the inputs that we got were comments from the
community.
We wrote a blog and did a content analysis of those.
That is what I'd like to start with because it gives
you a flavor for what these consistent concerns are.
So highlighting frequent comments, we heard ultimate
many times concerns that there's bias against junior
faculty members, women and those from under represented
groups, concerns that the current process is bias
towards senior mentors with extremely well funded labs,
concerns that institutional bias tilts towards large
universities with ample resources and the idea that the
rich get richer.
We also heard multiple reminders that junior faculty
can provide great mentorship.
In terms of people suggesting we could change this, the
request that we eliminate the requirement for
undergraduate grades.
Other comments we got a number of comments on relating
to how the science of fellowship reviews ought to be
evaluated.
We got pleas to educate reviewers to provide
constructive feedback and not to be overly harsh in the
comments with these very early career applicants.
And we got comments regarding the bias awareness
training that we were just talking about.
The idea that the bias awareness training ought to be
mandated for fellowship reviewers and the suggestion
that we think about creating a separate bias awareness
training module specifically aimed at fill low ships.
The concerns raised in these public comments sort of --
certainly resinated with the concerns that the working
group members had themselves.
They asked us to develop data to see if these were
borne out in review outcomes of fellowship
applications.
So we did.
I'm just going to highlight a few finding else from the
data.
So when you look, you do, indeed, see that applications
are highly concentrated in a small number of
institutions.
These data came from the 2021 council round.
We looked at all 6,676 applications at NIH reviewed
that year.
The figure shows you this.
Here you have bends that indicate how many applications
and institutions submitted during the year.
And the count is the number of institutions that fall
in that.
So if you look out here, starting at the right end of
this figure, you see that there are 15 schools that
each submitted over 100 applications, over 100
fellowship applications. If you look at the other end
of the distribution, you see there are over 100 schools
that submitted just 1 or 2 applications. There are
almost another 100 that submitted between 3 and 5
applications.
So it is highly skewed.
You might wonder whether the applications of the
schools that submit not so much applications are more
successful.
Yes, it's not black and white, but you do see that the
schools that submit the most applications do distinctly
better than applications that come from the schools
that submit fewer.
So down here on the bottom, the schools that submit
over 100 applications a year, they ended up scoring in
the high impact range.
The gray is the percent non discussed. You can see as
the number of applications submitted each your goes
down, so does the proportion that are highly scored and
the proportion that are non discussed goes up.
What about academic rank of the sponsor?
You see that review outcomes steadily improve as the
academic rank of the sponsor rises.
So let me direct your attention here to the top part of
this figure.
Look at these top four bars. If you start here at the
top, these are applications that came from assistant
professor sponsors, then associates, then professor and
then other.
In this grant, this figure, the applications that end
up scoring in the high impact range are shaded blue and
then here in gray, again, are the non discussed.
And you see that as you go from assistant to associate
to professor, the percents scoring in this high impact
range rises.
The percent discussed -- non discussed goes down.
This group, this other group does particularly well and
people wonder who that is.
These are deans, directors, investigators.
I think those must be like HMMI investigators.
These are senior folks.
And you can see that they do especially well.
Briefly, when you look at multiple sponsor
applications, you see the same sort of pattern. It is
somewhat accentuated.
Okay.
Then I just wanted to say briefly something about where
these review -- where the NIH review criteria come
from.
And what the limitations are in changing them.
So review criteria start in law. So the fellowship
criteria derive from this section of the code of
federal regulations, which says -- I'll just
paraphrase, but within the limits of funds available,
the Secretary shall may awards taking into
consideration, among other pertinent factors, these
four things.
So these are the four considerations on which we are to
evaluate applications.
One, the scientific, technical or educational merit of
the particular proposal.
Two, the availability of resources and facilities to
carry it out.
Three, if qualifications and experience of the
applicant.
And, four, the need for personnel in the subject area
of the proposed research or training.
So that is the law.
NIH has the authority and obligation to interpret the
law to turn it into criteria.
So that is what happens.
NIH takes the law, interprets it and this slide shows
you the current NIH criteria that represent NIH's
interpretation of the statute.
You can see that it is, indeed, an interpretation.
So the criteria fellowship applicant flows pretty
directly from this consideration three.
But if you look at the rest of the criteria, they're
clearly an interpretation.
They're not specifically mentioned in the
considerations of the law.
But they certainly flow from it.
So laws are really hard to change.
NIH can change its interpretation.
That's not necessarily easy, but it can be done.
And what the office of the general council says is that
the new -- any new interpretation must reflect the law.
So it has to derive from the Code of Federal
Regulations and it can be otherwise illegal.
So as long as you connect on the legislation and you're
not illegal, you're okay.
With that, I'm going to turn this over to you,
Elizabeth, and you can talk about the changes that the
group has contemplated.
So, Elizabeth, do you want to run your slides?
>> DR. ELIZABETH VILLA: Yes.
>> DR. BRUCE REED: So let me give up sharing here and
hopefully you can take over.
>> DR. ELIZABETH VILLA:
>> DR. HOPE CUMMINGS:?
>> DR. BRUCE REED: Yes.
>> DR. ELIZABETH VILLA: Here we go.
So the working group convened a few times.
And I think that the goals that we sort of set out for
ourselves is to say there are wonderful applicants out
there that may have, you know -- may not be one of the
15 or other expected institutions or don't have the
typical background or don't have enough access to
information to write those wonderful applications.
And so the idea is can we encourage them to apply and
can we revise the review criteria and other processes
to make sure they get a fair chance to have a
successful application.
So there's sort of five things that I'm going to talk
to you about today.
They kind of go in the order that they happen.
So first proposed changed to application materials.
And then the review criteria. A little bit about the
review process and then finally I'll tell you a little
bit of suggestions we have on those outreach and
reviewer training.
So the first are application materials.
Basically, the goal of this is to try to give a chance
for the applicants to present this to the reviewer.
We thought one of the most efficient ways was to change
the sketch and to be able to align the proposed
criteria to evaluate them.
So there are a few things incorporated into the
biosketch.
The first one is a personal statement and a personal
statement will have three parts.
The first one is their scientific interest.
Really, to answer why is this field of science
important and why are you selected it for your project?
So any sort of personal relationship between their
scientific interest and the project.
In goals -- and this is a really big one -- is what are
your career goals and how do those fellowship goals
relate to your career goals.
And here we want to make a point that there is nowhere
in the definition of an F application that states that
these successful applicants need to go to become
professors or scientists at certain institutions.
So we want to make sure that this is a place where they
can discuss other career goals and that the reviewers
will evaluate the entire proposal based on their career
goals.
Here they also have a chance to specify what difference
will the fellowship make in advancing your career.
So we're going to talk about this concept of the delta
today. We would like to encourage reviewers to
consider if getting this award will help someone take
off and have a wonderful scientific career.
And how that, you -- you know, what impact will the
award have on the fellow.
And last but not least, here is your fellowship
vision or goals.
That is what do you hope to achieve with this
fellowship and what knowledge, skills or expertise do
you hope to acquire.
And then what will this specific mentor and mentee team
help you to approach this goal.
So this is somewhere where they can bring in the
mentor.
I'll talk about this in the future.
But the idea is that we want the fellow to introduce
themselves to you and then in the second one will be
contributions to science.
Here in contributions to science, obviously if they
have papers, they can explain them, but we want to give
an open forum to say other things that they've done
that they consider contributions to science.
And then we have fellowship qualifications. We are
hoping that everybody will agree that grades for
undergrad are not needed, not for grad students and not
for post docs.
This is not a place to list their courses and grades,
also not the institutions they went to.
This is elsewhere.
This is really an opportunity to document experiences
that they have that they think will contribute to
qualify them for this proposal. This can be lab
rotations, conferences, internships, other appointments
or any life experience that I can contribute to their
professional development and will make them qualified
for these grants.
We are encouraging team to include outreach for
science, engagement or advocacy and other things that
might demonstrate their passion and commitment to a
successful scientific career.
And honors, which is pretty straightforward.
And last but not least, an optional part for special
circumstances. This can be pandemic effect, having
experienced harassment or any other kind of personal or
family difficulty etcetera.
Importantly, we want to allow that this last part, if
they don't want to include in their biosketch,
sometimes it might be more appropriate to allow the
institution a chair or someone else to add a letter
that might explain what these extenuating circumstances
have been.
So this is the application materials.
The second one is a pretty -- it seems drastic, but
it's actually a pretty good revision of the criteria.
So the current cry tear ya in the nutshell is on the
left.
What we want to do on the right is to take a away a --
this is a training fellowship. So the science needs to
be evaluated and the training needs to be evaluated and
the sponsor and the institutions are only important in
the context of these goals.
So we decided to take out the sponsor and the
institution as stand alone criteria. Because at
least -- I think for most of us, we are trained to
evaluate scientific achievements and environment in --
not in the concept of training.
So we -- and many of the people that gave input found
that a person who is a fantastic scientist might not be
a very good mentor, but not necessarily get low scores
when evaluated independently.
So we weaved all of these and ended up with three
proposed criteria.
The first one is focus on the applicant, the second on
the science and the third one on the training of the
proposal.
So to the applicant, this is the idea of casting a
wider net to find the commons in the rough.
So the idea for this will basically be first score the
research interests, professional goals and
qualifications of the applicant.
So basically the information including the biosketch.
In the second one, we will ask reviewers to try to not
focus only on their comments so far, but also
how likely it is that they will benefit from the
proposed fellowship.
Pay a lot of attention on what they talk about, their
determination, persistence, creativity and otherwise
potential to do incredibly well.
For science and scientific resources, the goal is to
eliminate sponsor and environment as stand alone
categories and weave them into the importance of both
the science and the training.
For the scientific part, we would like reviewers to
score the quality of the proposed science based on the
scientific rigger and feasibility of the proposed
project.
But we would like to introduce, again, like this delta,
which is how is the project serves the applicant
training goals and training needs.
So, again, we ask reviewers to think of the goals of
the scientist and try to figure out how this scientific
percentaging that they choose helps them advance
towards those goals.
The extent to which they have scientific resources
available to them realistically, not just that they
exist in the institution or that they have a fantastic
team of mentors, but that they are engaged in making it
into a successful application.
And then last but most least is to describe in detail
how the sponsored training team will provide this
expertise.
So both the resources available to them and sort of
like a human resources.
For the training plan it's a similar thing, but in this
instance, it's separate.
It's not only about the scientific project, but really
about the training program for the scientist.
And, again, evaluate a few considerations and the first
one is is the training program tailored to advance the
goals for the career that this person particularly
wants. How much of a difference this proposal does, so
this is the so-called delta.
If they have the proposal, do they have a larger
ability to accomplish these goals?
The extent to which the institutional training
resources are available to this candidate, not just
exist, so specifically saying, you know, they will
attend this workshops, they will learn these things a
little bit more specifically and, again, the plan for
how the sponsor and training team will provide the
teaching and mentorship that the applicant needs to
advance towards their career goals and to -- and the
goals of the proposal, of course.
So here is an opportunity to say if you're a junior
person, for example, what is your training plan, what
is your mentorship?
Here is your opportunity to describe how you're going
to be a great mentor for the student and how you'll
receive wonderful training and etcetera.
Last but not least, this was a common thing.
People say that if I don't have a number one, I'll
never get a fellowship or things like that.
This is an opportunity to say the need to demonstrate
there's sufficient resources to carry out the science
proposed, obviously e obviously, we know fellowships
only go towards the salary of the fellow.
They don't allow a lot of money -- a little money,
but not a lot of money to carry out this process.
Here is an opportunity for a junior person to say I
have enough starter funds to carry out this project or
there's money from different agencies or there's not a
lot of money required for these goals that target the
career goals of the applicant and the money is there
and then that is enough.
And it should not be -- you know, a good proposal does
not need to come from an incredibly largely funded lab.
Just, you know, sufficient and necessary for the
particular project.
All right.
So that is reviewing criteria.
We think there are other things that might help in the
review process, also held sort of evaluate all
candidates in the same way.
One of the things we learn training grounds already do
and are already sort of done in a way of the review
process is to cluster the applications and discuss them
separately.
So the minds are preparing like proposals and find the
best within that group.
There are two things that kept coming up.
One is early career sponsors feels like they don't have
a very good chance.
And institutions that are not research intensive
institutions.
One idea was to maybe separate research active versus
research intensive institutions and also, for example,
junior sponsor versus more senior sponsors being
discussed separately.
This is an ongoing discussion in the workgroup and we
would love your input in this.
Another suggestion change is to change the order in
which the information is presented.
Again, it's something that is being talked about in
general in the review process, which is that in the
biosketch, the first thing that you see is not the
institution sort of like the pedigree of the applicant.
It's something you see later in the application, but
maybe doesn't immediately make you think about sort of
that.
Last but very much not least, we thought of ideas to
improve outreach and reviewer training.
So I think you heard from Kristin the wonderful changes
her office is making and we would love to piggyback on
this.
The first to try to outreach schools that have fewer
resources.
In order-serving institutions or many institutions that
have a low frequency or no frequency of submitting F.
Applications, this would be good for partners with ICs.
Typically these kinds of things are done at scientific
conferences and things like that.
Some of us in particular think this is not good enough
because some people from some institutions don't have
access to travel and go to the scientific conferences.
So because of the online ability to serve multiple
institutions at the same time, this might be something
that CSR may be able to do, sort of offer online
workshops for a broad group of institutions to do this.
Obviously, the second one would be to have outreach
material that is accessible online to anyone.
And not necessarily through a workshop, but to try to
mimic or to say overcome the advantage that large
institutions have to provide, for example, successful
examples of previous applications or access to material
or things like that.
So this idea would be maybe having a website that are
almost like myth busters or frequently asked questions
that if you're concerned about this, these are ways to
address it.
These are successful outcomes, this is how the review
criteria would work.
If we change the review criteria, these might take some
time to do, but provide successful applications.
Show people that people from non conventional
backgrounds are successful and show them how they
became successful in doing these kinds of proposals
would be something that would be very nice to the
outreach.
There were other nice ideas about having people that
served on panels or chairs or things like that gives
sort of mentoring office hours or things like that
that might be able to review proposals and help people
develop them that might not have a direct ability to do
this within their institution.
And then for reviewer training, I think as you saw, a
lot of people thought that specialized reviewer
training for fellowships would be a very nice thing to
develop.
Again, evaluate the application as training merits, not
on sponsor reputation.
It's something that came over very much.
Be very careful of hurtful and discouraging language.
This is something that I think a lot of us have
experienced, that person who got their fellowship
reviews and they were not constructive and it really
kind of makes a dent in their enthusiasm for science
that I think just doesn't need to be there.
So I think it seems like it is a good thing to remind
us of that.
Make sure that reviewers understand that if someone
says that they don't want to become a PI in the image
of their sponsor is okay.
There are a lot of wonderful scientists needed in all
sorts of life and particularly these fellow ships are
for those that have a future career in biomedic
research.
There's a lot of flavors.
Sorry.
I thought I had one more. Apparently not.
Last, I want to open the discussion to council. We
would like your discussion and reactions, especially
regarding these ideas.
Eliminate the sponsor, collaborators and institutional
requirement in the training criteria, explicitly allow
a wider range of career paths.
Have reviewers consider "the delta," how much of a
difference this would make in the fellowship of their
career.
Eliminate grades as indicators of qualifications,
encourage a statement of applicant qualifications that
extends beyond course, grades and publications. And
then allow an optional statement of special
circumstances and optionally have this to come if it is
school.
That is it. Okay. I'll put that back so you can see
If you want to unshare, we'll open it up for discussion
at this point.
Mark.
>> DR. MARK PEIFER: Could she lead the slide up while
we have a summary of that last --
>> DR. BRUCE REED: Sure.
Sorry.
>> DR. ELIZABETH VILLA: There you go.
>> DR. MARK PEIFER: I think everyone will agree with
some of these things like eliminating grades.
But I want to focus on the bigger picture thing.
I want super excited about this.
I think this potentially will have as much or more
impact than the discussion that we had of review
criteria overall.
And, you know, the big picture, I think we all realize,
right, that the fellowship programs, to a large extent,
have become an opportunity for people who have had
privilege to get more. That would fit my own career,
right?
I had a -- and NSF graduate fellowship.
Once you get on that ladder and you're at the right
institution, working for the right person, then all of
these things become much easier for you to get.
Again, not to diminish the -- you know, the work of all
those people who are in labs like that who write a
fantastic proposal.
So for me, the most important things I see here are --
and, again, completely in concert with what we did with
respect to overall review criteria, greatly reduce the
constitutional environment piece except as key parts of
the training plan.
Someone might be a Nobel laureate, but if they never
see their graduate student or post doc, they're not
providing a training environment to somebody doing
their very best.
And I really like your description of "the delta," the
idea that we're -- you know, the idea that taking
somebody with emotional and moving them up, I think
that's a place where these kinds of programs can have
the -- by far, the most impact.
I'm very excited about it.
>> DR. KAREN ANDERSON: I just cannot say how much this
could impact the diversity in S.T.E.M. moving forward
and this just really touches on so many areas that
might students have had, these comments, these exact
comments on multiple different trainee grants.
And in many of press approaches, we're in a Hispanic
serving institution, this culture shift in
transformative change, I think, can really impact just
a tremendous amount of students across the country and
I'm really looking forward to this. I have no specific
comments on any of this, other than I just want to say
from the bottom of my heart, thank you to the working
group and to all -- to the incredibly thoughtful
approach that you've taken to this.
>> DR. BRUCE REED: Thanks, Karen.
Tonya.
>> DR. TONYA PALERMO: Great job, Bruce and Elizabeth
and the workgroup, as well.
I wanted to ask you a little bit more about the
interesting component that you added around evaluating
this delta or the difference the fellowship makes.
I'm not sure how the applicant will write about this in
the fellowship and how that would be reviewed.
I'm trying to understand how that could be
evaluated, I guess, with less judgment because it seems
like a potentially difficult thing to vault.
>> DR. ELIZABETH VILLA: Bruce, do you want to try or --
>> DR. BRUCE REED: Why don't you start.
>> DR. ELIZABETH VILLA: Okay.
That sounds good.
So, first of all, I forgot to say that it was an
absolute pleasure to work with the workgroup.
I think they were very sort of like strongly
opinionated group which was great and finding consensus
was not easy.
Tonya, to your point, I completely agree.
I think that it is not trivial to establish and it's
probably on the fellow and the -- and the -- or the
applicant and the sponsor to try to explain these
things.
Like if I get this, putting it bluntly, it gets me on
the train that Mark just described.
We came to this and I go to the same place, because
ideally we would hope -- again, this is a program
decision -- that there would be different pools of
applicants where you could -- you know, people that are
extraordinary will write flawless proposals would
presumably still get these fellowships.
They still deserve it and they're great.
And then to give an opportunity to expand that pool to
help people get on the right track to have a
presentation in future generations is what we were
trying to do. It is not a trivial thing to include at
the reviewer level, but this is what we're trying.
So, yeah, I think mentioning if I had this proposal, I
would have access to other trainings that I couldn't
otherwise because my adviser might use those funds to
do this or to do that.
It's not -- yeah, so I think that the collective brain
of council and others that might be listening to help
us materialize this would be great.
As I said, one thing that would be great is to try to,
in the future, sort of try to encourage program to
have -- you know, there are instances in which you have
the same RFA, but different sort of pools of money that
allow you to not only cluster the applications and
discussion, but actually assign them
differently.
This is not something that would happen here, but maybe
in the future it could.
>> DR. BRUCE REED: I would say evaluating these
candidates are challenging.
I think the delta came from both directions. Sometimes
you'll see an application and it's a fabulous
application. And you end up thinking even if the
person doesn't get this fellowship, they're going to do
great. They are in a fabulous environment. They are
going to do great. You see other people for whom
you're not so sure that that is true.
They seem well qualified.
They seem to have great potential.
But they're not maybe in as well a resourced place.
They don't maybe have the same connections.
And you think if they don't get this, yeah, it probably
is going to make a difference one way or the other, a
lot of difference in their career.
So part of this is sort of a -- I guess a philosophical
thing of where NIH should put its resources and that's
clearly not just a with you decision.
And then, of course, there is a challenging decision of
how do you evaluate on who those people are.
Michelle.
>> DR. MICHELLE JANELSINS: Great presentation,
Elizabeth and Bruce.
Really liked a lot of it.
Definitely support the getting rid of the grade
requirement. You know, somebody got into grad school,
then that's great and they can say whatever they would
like about their grades if they want in their personal
statement.
But that is one I strongly feel needs to go.
And then I guess, you know, my other thought -- I have
just one comment, but also a question.
One comment would be -- and I really like this idea of
evening the playing field and creating diversity of
scientists in chosen profession.
And so then it's kind of like a little bit like a
marketing thing.
How can we encourage and get the word out to those in
all different types of colleges, you know, like through
organizations.
I certainly didn't know necessarily about an F
application -- I mean, I had an F award, but I didn't
know about it until I was in that school, in that grad
school.
So just getting the word out as a possibility just to
something to think about.
And then my question, I do think that it's a really
intriguing idea to think about similar, like, doing ESI
for R01 review together and then the more experienced
seasoned types of applications together or ESI and non
ESI, I guess. Would you review the assistant
professors together and everybody else or would you do
tenured, non tenured?
What was the thought about how things might be split if
there was a dedicated discussion area to an earlier
career mentor, for example?
>> DR. ELIZABETH VILLA: So there was a lot of
discussion about it.
We did not reach a consensus partially because some
people thought, well, you know, maybe you're lucky and
you're not ESI any more, but it's your first fellowship
application or, you know, where to draw the line,
assistant versus non.
So building a new category, apparently, is not a
trivial thing to do within the databases.
A discussion was made of, like, only if it's the first
time you -- like if you've never had an F fellow in
your lab, you would be a newbie and this would allow to
have people from other areas, if their funding is not
NIH, their considered newbie only once.
So they can't populate their labs with 40 fellows
because they're only senior not typically NIH.
But I think some people prefer to have only really
newbie sponsors.
So we haven't tuned in on one and that it would be
something left to get input from you.
It is the same thing with research active versus
research intensive institutions when people thought
idea would be one thing but a lot of people say there's
people that are a non idea stage.
There's plenty of institutions in California or
Massachusetts that are not research intensive that
would also benefit from being discussed separately.
So these are things that we're still maybe opening
discussion and to feedback from you and everyone else
and how to implement that. We would like the concept
and we're still in discussions about how to implement
it.
>> DR. NONI BYRNES: There's people in line.
I do want to say a couple of comments at some point,
but please go ahead and get the people in line first.
>> DR. BRUCE REED: Let's go Leo, Jinming and then I'll
go to you, Noni.
Leo.
>> DR. LEOPOLDO CABASSA: I really appreciate the
discussion on the work.
One I guess more thinking about the focus of the
sponsor, and it's not -- I get the -- I totally get the
analysis that you have done here, which is very
telling.
But I've seen it both ways.
I've seen it where too senior is great and too junior
is also great and there's a catch-22 that happens
sometimes.
It's not the elimination of the sponsor, but maybe
around the delta that you're talking about about what
is the training plan, what is the engagement.
>> DR. ELIZABETH VILLA: Right.
>> DR. LEOPOLDO CABASSA: So I think some guidance
around that would be very important and access to that.
And then I do want to say as a directive of T-32, this
is great.
I appreciate the changes in the F-31.
But there are unintended consequences here that could
happen.
So it's thinking through how this will play out of the
training grounds because, you know, T-32s want F-31s
and they want KO awards.
So I think a discussion needs to be how this influences
the ecology of training within NIH.
>> DR. ELIZABETH VILLA: So we had -- we had people
from program specializing in T-32s and people that are
directors of T-32s.
I mean, everybody is worried every time you change a
criteria there are unintended consequences.
That's why we keep revising criteria forever and ever.
But we engaged -- besides at least me personally, but I
think a lot of people engage people that are interested
in these outcomes and everybody seemed to think that
this would be pretty -- a pretty reasonably good
outcome.
Again, ideally maybe there would be, you know, separate
and clarify and things like that.
But, you know, separate pools for the delta and non
delta.
But, yeah, I think if you have any specific thoughts of
what outcomes -- unintentional outcomes in the future,
just, yeah, please --
>> DR. LEOPOLDO CABASSA: And I think this is great.
What I was saying is this is an opportunity to think
through the applicants of T-32 because this opens the
pool to a whole other applicant that as T.-32 you might
be like I don't want to bring that person in because
two or three years down the road the Fs will be
difficult because that person doesn't have the right
training.
So this opens that pool for providing opportunities.
>> DR. ELIZABETH VILLA: That is great.
>> DR. LEOPOLDO CABASSA: So I think that's great on
thing that through >> DR. ELIZABETH VILLA: And it's
making sure that everybody has the opportunity to
describe why they would be a good sponsor and just
being a fantastic institution and being a fantastic and
accomplished scientist does not immediately make you a
good sponsor.
I think it's what we are trying to say, that everybody
needs to -- everybody has, you know, whatever metrics
they want to demonstrate that makes a great sponsors,
they have a chance to do it, but it's not -- yeah, I
think that was -- that's what I'm -- that's what we're
trying to accomplish, anyway.
And I think by weaving it as opposed to having it
separate, it's easier for the reviewers not to do the
same thing which is in the review train, oh, it's Dr.
Fauci.
Hundreds of people have come out of that lab and become
successful.
That is not necessarily a given and that's what we're
trying to capture.
>> DR. BRUCE REED: So I'm going to go the Jinming.
And then to Noni.
We're tight on time.
I can let this run.
I acknowledge that it's 5:00.
This is a scheduled end so if you have a hard stop and
have to, that's unfortunate but I understand that.
But I will let the discussion run for a bit here.
Jinming.
>> DR. JINMING GAO: Thank you, Bruce. Thank you,
Elizabeth were for doing this.
I can imagine the heated probably discussion in the
panel.
I guess we are all not for giving free passes and how
do we provide equal opportunities to the candidates.
But I think we have to set up the guiding principles so
they don't lose equality or meritocracy in the
evaluation process.
And then there are sort of a system-based correction.
I wonder that for all of us working am deem ya, it's
not necessarily the most attractive career path.
I'm looking at my two boys. I don't think either of
them are serious think thinking about being a professor
when they grow up.
So I think this is sort of a threat to our profession
and systemically how do we, you know, raise the level
of the pay line so maybe diversifying the career path
and making it more enticing for the young people to
think about science and academic careers as something
exciting.
I wonder whether from the systems standpoint is
something helpful to get more talents into the field.
>> DR. ELIZABETH VILLA: I think you are absolutely
right that increasing the pie is always a good solution
if you can do it.
I'm not entirely sure it's feasible.
But I do want to say one thing about what you just
said.
And that is that there's actually a paper that shows of
F applicants, only a small fraction go to get a narrow
ride. Which is preoccupying itself.
And also we need fantastic scientists everywhere.
If you don't think that there's amazingly good careers
in policy, junior editor, you know, working in the lab
and all sorts of other things, I think that, you know,
we need to think about how we're presenting other
career goals because I think they're fantastic.
I don't know that my daughter is going to have a
professor, but I think there's ways to be scientists
that are very varied. There's a small sliver of
trainees that become professors.
In any meaningful way.
Respectfully.
I think we need to also think about maybe preparing
them for amazing careers that are alternative to ours
and, you know, because it's -- this is what the job
market is like, anyway.
So it is important to be supportive of other career
paths, I think.
>> DR. BRUCE REED: Noni and then I'll come back to
you, Mark. I see your hand up.
>> DR. NONI BYRNES: First, I want to say thank you so
much.
This is a tremendous amount of work.
It's very thoughtful and I really appreciate how you
took all of the information that we got and came up
with some really fantastic directions to go in.
This is an interim report so I just want to provide a
little bit of guidance moving forward before we get to
the final.
One is in -- you saw the scope of applications.
The more effective they are, the more complex. Within
the NIH, there's a tendency to make things complex
because you want to address everything that you think.
And I think that doesn't come out effective at the end
of the review process.
Just the sheer number of reviewers we have to train,
they have to behave the way you want them to behave in
an ideal world.
They don't.
So the simpler you make, the less complicated the
process, the easier it is. I would suggest you look t
different directions you're proposing and separate out
what is something that is very much something we can
actually tackle and review and what is something that
is kind of a broad -- really important, but broader NIH
direction. Which we can capture in the report, but we
both -- outreach, for example, the fellowship
applicants, especially in certain scientific areas.
There's some scientific areas that spent a lot of
fellowships and others that submit very few.
The ones that submit very few have a lot of URMs in
those fields.
I think thinking about that and developing those
recommendations separately would be helpful to us.
So I wanted to put that out there as a direction moving
forward.
So thank you very much.
>> DR. ELIZABETH VILLA: Thank you.
And thanks, everybody, for your input.
>> DR. BRUCE REED: And Mark.
>> DR. MARK PEIFER: How would you measure the delta?
Having reviewed and advised folks for the HHMI Gilliam
program, they have developed ways to measure that.
The way they set up their applications involves people
telling their story and that gives you an opportunity
to see already the delta that somebody has come
through, where have they come from to get to where they
are today?
And that kind of information gave me, when I was
reviewing those applications, some confidence in
predicting if I give this person an opportunity, where
are they going to go with it?
That kind of resilience and ability to succeed, I
think, is a scorable criterion. And I do have to
pushback on something I just heard.
We just came back to the word meritocracy.
I'm sorry, but our business is not a meritocracy. We
can pretend it is.
We can pretend everyone has an equal opportunity, but
we just saw the data, you know? 15 institutions with
30% of all the applications and an even larger for
action of the successful applications.
If we try to convince ourselves that by changing the
criteria we're going to damage the meritocracy, well, I
find that disturbing.
I'm sorry, but I just had to say that.
>> DR. BRUCE REED: Jinming.
>> DR. JINMING GAO: If I may quickly respond, I feel
we're trying to correct the bias towards the big
institutions.
I think giving people the free right.
But I think we do have the set up the metrics, though,
to really not penalize, I feel, the candidates because
they get to the bigger institutions.
>> DR. ELIZABETH VILLA: Yeah.
But I specifically said that, right? Anybody equally
can just -- you know, there is no penalization, there is
no punishing for being successful.
>> DR. JINMING GAO: No, just the comments from Mark is
where I guess I'm coming from.
>> DR. BRUCE REED: Yeah.
I want to echo what Elizabeth said, which is that,
you know, I think the fundamental focus of the working
group was on identifying strong candidates, how do you
do this?
How do you do it fairly?
How do you really give opportunity across the board and
open it up to people who are in places where there is
less cumulative knowledge, where there are less
resources, where there's less experience, but
nonetheless there may be fabulous potential.
And that has been what the group is struggling with.
Okay. Thank you, everybody.
It's 5:05.
This has been a really -- a very interesting
discussion, an interesting meeting. Thank you all.
I'll just say on behalf of the working group, if you
have other ideas, feel free to shoot me or Elizabeth,
anybody in the group an email.
We really do want -- would like to continue the
discussion and, of course, we will be coming back to
council at another time with more refined set of ideas.
So council, I'll sorry I left you no time for open
discussion.
>> DR. NONI BYRNES: We've had lots of open discussions
across.
>> DR. BRUCE REED: So thanks for everything.
And with that, we will adjourn for today.
>> DR. NONI BYRNES: Thank you, folks.
>> DR. ELIZABETH VILLA: Thanks, everybody.
>> Thank you.