Therapeutic Product Immunogenicity Community

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

Hi there,

One of the assumptions for the NC is that it is representive of the population of interest. If you are seeing the study samples running (much) lower the the validation NC, then I would strongly recommend identifying/creating and new NC pool that is closer to the study population. This might require running numerous individual samples to identify those that are most representative of the study population before pooling (commercial pools might be skewed by a small number of individuals)

Whilst it is theoretically possible to run analysis with a negative cut-point factor, it could be questioned whether the day-to-day variation in signal of the NC tracks with the day-to-day variation in the study samples.

On the point of titering, if you are not able to reproducibly dilute samples below the screening cut-point (SCP), it is common practice to set a titer cut-point (TCP) at the 99th or 99.9th percentile to move out of the noise. This is described in both the Devanarayan et al 2017 paper and the Anti‑Drug Antibody Validation Testing and Reporting Harmonization paper (Myler et al 2022). 

Thanks,

Rob

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

Dear Anonymous,

                I'm afraid I don't have any advice for you that wouldn't require lots of tedious work. However, your conundrum underscores the weakness of the cut point idea as some threshold response that delineates ADA "negative" from "positive". There are a couple of reasons for this. First, a typical bridging assay format does not detect ADA – it detects species that can bind multiple molecules of the drug. It could be the drug target, drug target in complex with some other protein(s) or it could be the ADA. And you don't need to know if the cut point is 1.06 or 1.60 to detect the telltale signs of ADA presence. All it takes is to take a look at the time course of S/N response in your ADA assay and its relation to PK and PD measurements. It's rather preposterous to call a sample ADA-positive just because it gives a response ≥ cut point. Cut point has nothing to do with ADA; it's just part of a statistical description of drug-naïve (and presumably ADA-negative) population.

Which leads us to the second weakness of the cut point approach. Cut points are determined in drug-naïve commercial samples which are very different from the study population. Study samples are not drug-naïve, they contain the drug and they may also contain ADA. Additionally, if the drug has the desired biological activity, the study population is unlike any other; these are people who were previously sick and now are better because of the treatment. In other words, study samples are very different from anything you can buy from a commercial source. Perhaps the best example of the disconnect between the validation cut point and the study samples is the cut point for nAb assays. NAb cut point is determined in a drug naïve population, but the samples tested in the nAb assay come from a subset of patients in the drug-treated population who are ADA-positive. Believing that validation cut point determined in drug-naïve samples is relevant to drug-treated and ADA-positive samples is just wishful thinking on our part.

I could ramble longer, but I invite you and everyone in the community to re-think our approach to immunogenicity testing. We have lots of new data on ADA and its impact since the first white papers were published back in 2004 and 2008. It's time to examine and draw lessons from about 20 years of data. It seems to me that during assay validation we do a lot of work for little return, while in study we tend to waste patients' samples by subjecting them to multiple tests of low or no value. 

Cheers!

Robert

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

@Robert, you make some great points, so I can't help chiming in here.  The response I hear most when I challenge the need for cut points is that we 'need to know what is positive or not'.  But, to your point, the cut point is a statistical calculation, not an arbiter of clinical significance.  This is where I think we have fallen down a bit of a rabbit hole chasing what we can theoretically measure vs. what we originally intended to measure.  As you point out, looking at the time course of S/N and its relationship with PK, PD and safety/clinical measures will identify relevant responses, while avoiding the swirl that often accompanies 'high' incidence (of clinically irrelevant responses).

In every instance to date where I have reanalyzed my ADA screening data in this fashion I have either come to similar or more insightful conclusions (mainly the latter) as achieved with the current paradigm.  This analysis is faster, less expensive and provides greater depth of information.  While it can be difficult to effect a paradigm shift, I try to remind myself that this is why I chose science - because the answers change as we build larger data sets and iterate our thinking.

Dear Anonymous,

                I'm afraid I don't have any advice for you that wouldn't require lots of tedious work. However, your conundrum underscores the weakness of the cut point idea as some threshold response that delineates ADA "negative" from "positive". There are a couple of reasons for this. First, a typical bridging assay format does not detect ADA – it detects species that can bind multiple molecules of the drug. It could be the drug target, drug target in complex with some other protein(s) or it could be the ADA. And you don't need to know if the cut point is 1.06 or 1.60 to detect the telltale signs of ADA presence. All it takes is to take a look at the time course of S/N response in your ADA assay and its relation to PK and PD measurements. It's rather preposterous to call a sample ADA-positive just because it gives a response ≥ cut point. Cut point has nothing to do with ADA; it's just part of a statistical description of drug-naïve (and presumably ADA-negative) population.

Which leads us to the second weakness of the cut point approach. Cut points are determined in drug-naïve commercial samples which are very different from the study population. Study samples are not drug-naïve, they contain the drug and they may also contain ADA. Additionally, if the drug has the desired biological activity, the study population is unlike any other; these are people who were previously sick and now are better because of the treatment. In other words, study samples are very different from anything you can buy from a commercial source. Perhaps the best example of the disconnect between the validation cut point and the study samples is the cut point for nAb assays. NAb cut point is determined in a drug naïve population, but the samples tested in the nAb assay come from a subset of patients in the drug-treated population who are ADA-positive. Believing that validation cut point determined in drug-naïve samples is relevant to drug-treated and ADA-positive samples is just wishful thinking on our part.

I could ramble longer, but I invite you and everyone in the community to re-think our approach to immunogenicity testing. We have lots of new data on ADA and its impact since the first white papers were published back in 2004 and 2008. It's time to examine and draw lessons from about 20 years of data. It seems to me that during assay validation we do a lot of work for little return, while in study we tend to waste patients' samples by subjecting them to multiple tests of low or no value. 

Cheers!

Robert

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

Hi @Lauren. I sort of expected that you might chime in 😊, and we do need a community discussion on various forums. I understand people may be concerned that discarding cut point would mean that we can no longer tell what's positive vs negative. I agree that you know ADA when you see it, but when programmers program tables/figures/listings, what should they look at to "see" ADA as well? Examination of ADA time profiles is very informative on the subject level, but on the population level we still need to summarize immunogenicity results from hundreds or thousands of patients in language that can fit in a single paragraph on the prescribing label. I tried to put some ideas together, but at this point they are too garbled to share even this informal setting.

Need mare data to play with.

@Robert, you make some great points, so I can't help chiming in here.  The response I hear most when I challenge the need for cut points is that we 'need to know what is positive or not'.  But, to your point, the cut point is a statistical calculation, not an arbiter of clinical significance.  This is where I think we have fallen down a bit of a rabbit hole chasing what we can theoretically measure vs. what we originally intended to measure.  As you point out, looking at the time course of S/N and its relationship with PK, PD and safety/clinical measures will identify relevant responses, while avoiding the swirl that often accompanies 'high' incidence (of clinically irrelevant responses).

In every instance to date where I have reanalyzed my ADA screening data in this fashion I have either come to similar or more insightful conclusions (mainly the latter) as achieved with the current paradigm.  This analysis is faster, less expensive and provides greater depth of information.  While it can be difficult to effect a paradigm shift, I try to remind myself that this is why I chose science - because the answers change as we build larger data sets and iterate our thinking.

Dear Anonymous,

                I'm afraid I don't have any advice for you that wouldn't require lots of tedious work. However, your conundrum underscores the weakness of the cut point idea as some threshold response that delineates ADA "negative" from "positive". There are a couple of reasons for this. First, a typical bridging assay format does not detect ADA – it detects species that can bind multiple molecules of the drug. It could be the drug target, drug target in complex with some other protein(s) or it could be the ADA. And you don't need to know if the cut point is 1.06 or 1.60 to detect the telltale signs of ADA presence. All it takes is to take a look at the time course of S/N response in your ADA assay and its relation to PK and PD measurements. It's rather preposterous to call a sample ADA-positive just because it gives a response ≥ cut point. Cut point has nothing to do with ADA; it's just part of a statistical description of drug-naïve (and presumably ADA-negative) population.

Which leads us to the second weakness of the cut point approach. Cut points are determined in drug-naïve commercial samples which are very different from the study population. Study samples are not drug-naïve, they contain the drug and they may also contain ADA. Additionally, if the drug has the desired biological activity, the study population is unlike any other; these are people who were previously sick and now are better because of the treatment. In other words, study samples are very different from anything you can buy from a commercial source. Perhaps the best example of the disconnect between the validation cut point and the study samples is the cut point for nAb assays. NAb cut point is determined in a drug naïve population, but the samples tested in the nAb assay come from a subset of patients in the drug-treated population who are ADA-positive. Believing that validation cut point determined in drug-naïve samples is relevant to drug-treated and ADA-positive samples is just wishful thinking on our part.

I could ramble longer, but I invite you and everyone in the community to re-think our approach to immunogenicity testing. We have lots of new data on ADA and its impact since the first white papers were published back in 2004 and 2008. It's time to examine and draw lessons from about 20 years of data. It seems to me that during assay validation we do a lot of work for little return, while in study we tend to waste patients' samples by subjecting them to multiple tests of low or no value. 

Cheers!

Robert

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>

@Robert, this is another question I hear frequently.  At the end of the day (or maybe I should say at the beginning of the day?) immunogenicity is fundamentally a biomarker - it is a biological response to a therapeutic intervention.  The context of use is to determine what level of response has clinical relevance.  We manage biomarker data sets and describe and summarize them in clinical reports all the time. The beauty of no cut point is you can use all the data to inform your analysis.  The placebo population helps you define the true biological noise and responses above that noise can be parsed as you see fit.  For biomarkers, people often start by looking at quartiles or tertiles of the response magnitudes and look to correlate those groups with clinical impact, but in truth one could look at any number of potential cut offs and re-evaluate the data.  Does this mean you can't just call POS or NEG out of the gate - yes. (But we know that the POS/NEG call is frequently misleading and inflates concerns over incidence without impact).  Conversely, does  it allow you to use a data-driven approach to identify meaningful responses - also yes. If one proposed to regulators that they were going to do a statistical calculation on the signal from known negative samples to identify a relevant biomarker, they'd get nowhere.  Identifying clinical relevance requires fully analyzing the data set. Cut points simply cause us to put blinders on, look at only a subset of the data, and then over-emphasize the importance of the POS data regardless of magnitude.  My recent presentation at EBF included multiple case studies where the CP caused important data to be overlooked and erroneous conclusions about the overall study to be drawn, while also incurring unnecessary time, money and resource.

So, in short, the data need to be handled as we do for any other continuous response/impact analysis as we do for PD and other biomarkers.  It's different from how we have reported and summarized immunogenicity data in the past, but it is not a process with which we are unfamiliar.  Ultimately, under the current paradigm, we are trying to get to clinical impact, which is typically described as subjects with titers > x had impact y.  Here we have an opportunity to get there more quickly and with more granularity, with a conclusion instead being subjects with S/N >x had impact y.

But, in so many cases with modern drugs, we don't see any clinically relevant ADA or by bucketing POS vs NEG we miss insights because meaningful responses in a few subjects get lost in the noise of low level, irrelevant responses. Additionally, titer imprecision adds noise to the analysis.  We adopted the current paradigm for all the right reasons at the time - we were in new territory and didn't have data.  Now we are sitting on over two decades of data and in that same timeframe we have learned a lot about biomarker analysis.  Armed with that knowledge, I believe we can build a path forward that reduces complexity and confusion and delivers greater clarity.

Hi @Lauren. I sort of expected that you might chime in 😊, and we do need a community discussion on various forums. I understand people may be concerned that discarding cut point would mean that we can no longer tell what's positive vs negative. I agree that you know ADA when you see it, but when programmers program tables/figures/listings, what should they look at to "see" ADA as well? Examination of ADA time profiles is very informative on the subject level, but on the population level we still need to summarize immunogenicity results from hundreds or thousands of patients in language that can fit in a single paragraph on the prescribing label. I tried to put some ideas together, but at this point they are too garbled to share even this informal setting.

Need mare data to play with.

@Robert, you make some great points, so I can't help chiming in here.  The response I hear most when I challenge the need for cut points is that we 'need to know what is positive or not'.  But, to your point, the cut point is a statistical calculation, not an arbiter of clinical significance.  This is where I think we have fallen down a bit of a rabbit hole chasing what we can theoretically measure vs. what we originally intended to measure.  As you point out, looking at the time course of S/N and its relationship with PK, PD and safety/clinical measures will identify relevant responses, while avoiding the swirl that often accompanies 'high' incidence (of clinically irrelevant responses).

In every instance to date where I have reanalyzed my ADA screening data in this fashion I have either come to similar or more insightful conclusions (mainly the latter) as achieved with the current paradigm.  This analysis is faster, less expensive and provides greater depth of information.  While it can be difficult to effect a paradigm shift, I try to remind myself that this is why I chose science - because the answers change as we build larger data sets and iterate our thinking.

Dear Anonymous,

                I'm afraid I don't have any advice for you that wouldn't require lots of tedious work. However, your conundrum underscores the weakness of the cut point idea as some threshold response that delineates ADA "negative" from "positive". There are a couple of reasons for this. First, a typical bridging assay format does not detect ADA – it detects species that can bind multiple molecules of the drug. It could be the drug target, drug target in complex with some other protein(s) or it could be the ADA. And you don't need to know if the cut point is 1.06 or 1.60 to detect the telltale signs of ADA presence. All it takes is to take a look at the time course of S/N response in your ADA assay and its relation to PK and PD measurements. It's rather preposterous to call a sample ADA-positive just because it gives a response ≥ cut point. Cut point has nothing to do with ADA; it's just part of a statistical description of drug-naïve (and presumably ADA-negative) population.

Which leads us to the second weakness of the cut point approach. Cut points are determined in drug-naïve commercial samples which are very different from the study population. Study samples are not drug-naïve, they contain the drug and they may also contain ADA. Additionally, if the drug has the desired biological activity, the study population is unlike any other; these are people who were previously sick and now are better because of the treatment. In other words, study samples are very different from anything you can buy from a commercial source. Perhaps the best example of the disconnect between the validation cut point and the study samples is the cut point for nAb assays. NAb cut point is determined in a drug naïve population, but the samples tested in the nAb assay come from a subset of patients in the drug-treated population who are ADA-positive. Believing that validation cut point determined in drug-naïve samples is relevant to drug-treated and ADA-positive samples is just wishful thinking on our part.

I could ramble longer, but I invite you and everyone in the community to re-think our approach to immunogenicity testing. We have lots of new data on ADA and its impact since the first white papers were published back in 2004 and 2008. It's time to examine and draw lessons from about 20 years of data. It seems to me that during assay validation we do a lot of work for little return, while in study we tend to waste patients' samples by subjecting them to multiple tests of low or no value. 

Cheers!

Robert

Hello all,

I recently ran into an issue where the in-study and validated cut points were found to be different. Both the variances (by Levene's test) and means (By T-test) for the study population were found to be different than the population used for the cut point calculation during validation. Additionally, the in-study cut point was calculated to be less than 1. 

 <u5:p></u5:p>

So, following the Fig 8. flow chart from Devanarayan et al. (2017) (screenshot attached), we are in the bottom right section..<u5:p></u5:p>

• Option 2 does not work for us as the new cut point will make the NCs screen positive and cause plate failure.<u5:p></u5:p>
• This leaves us with Option 1. We can screen different individual samples to confirm a lower response in the assay than the current NC pool and make a pool with that. 

To reduce additional validation work, I was wondering if there is any precedence of using two different cut points for the same plate. That is, the validation cut point for the controls on the plate, and the in-study cut point for the samples? Given that this assay would be used for multiple indications, The goal is that if the other indications also seem different, l do not have to keep changing NCs again and again. One issue I do see from not changing the NC (in this specific case) is with titration runs, where I will have to dilute the samples with NC itself, and the signals might never cross the titer cut point.

Your advice on how you have dealt with issues like this would be highly appreciated.

Thank you

<u5:p></u5:p>