Pia Abola: Welcome to Countering COVID a podcast by LGC Biosearch Technologies, where we speak with clinical lab scientists and industry experts to understand their perspectives on the current and future state of SARS-CoV-2 testing. My name is Pia Abola, and I'll be your host. On this episode we'll take a closer look at the SARS-CoV-2 testing workflow, focusing on how to tackle false positives and false negatives. Here to discuss this topic we have Jim Huggett. Jim Huggett: Hi. Pia Abola: Jim is a scientist overseeing nucleic acid metrology research at the UK National Measurement Laboratory, which is hosted at LGC. Jim also has a joint appointment at the University of Surrey. We also have Tania Nolan. Tania Nolan: Hi. Pia Abola: Tania is a scientist with over two decades of experience in the application of qPCR technology and qPCR assay development. Tania is a consultant to leading life science companies, including LGC Biosearch Technologies. Lastly, we have James Grayson. James Grayson: Hey Pia. Pia Abola: James is an expert in the automation of molecular diagnostics workflows, and a field application scientist with UgenTec. A company that provides software that automates and standardizes the entire qPCR assay and analysis workflow for molecular diagnostics. So thank you everybody for joining us. And I would like to start by talking about the definition of a positive qPCR result. Because before we can talk about false positives and false negatives, I think it's really important to define what makes a positive? Jim Huggett: A positive, so we're talking about an analytical positive here to be clear, not a clinical decision based on this, just the performance of the experiments so you're an analytical positive. You would basically have the change in fluorescence that follows a particular shape during a PCR reaction, you get an increase in fluorescence as the cycles of the PCR continued. And that would generally was a big discussion here, depend on how much nucleic acid was there at the beginning, by which I mean, you have more nucleic acid it takes fewer cycles to pass a particular threshold, quantification cycle than if you have less nucleic acid at the beginning. And that would require more cycles to reach the same point. And that essentially talks about a positive reaction when you're performing qPCR. Pia Abola: So if you have no DNA in the sample, then you will never get that fluorescence? Jim Huggett: So there are circumstances where that will be the case, but crucially in the context of this is that for SARS-CoV-2 diagnosis where you a specific probe, there really should not be any signal that you would accept as a background signal, really. There's reasons for false positives, but a background signal that you might measure as you might in other examples of analytical chemistry, where you would have two or three standard deviations above which you give a result. This should not be the case in a qPCR targeting SARS-CoV-2, and so there really should be nothing ideally. You're aiming for no signal at all. And it's quite unique in an analytical scenario to have nothing as the result for a negative, but that is essentially where we should be aiming for. Pia Abola: Tania, do you want to talk about where a false positive might occur in the workflow? Tania Nolan: So, absolutely. And I think really to reiterate part of what Jim said, even to begin with, unless your negative control is completely negative than anything that you see in the positive you need to start asking questions about because both of those controls are important to be considered together. So where might you get false positives from? Well, it could be quite as simply as the end-user carrying target from a previous reaction on their clothing or gloves or whatever and transferring it. So person transfer, and then you would end up with a false positive. It could be as we've seen in the recent pandemic that during the development of assays. So before they're being used in a clinical situation, somebody has to go through the process of designing those assays and verifying that they actually work. Tania Nolan: Well, what do you use to verify that these assays work and to determine how sensitive they are. Well, you need some kind of target. Ideally, that needs to be something that's fairly standardized. So one very simple solution has been to request synthetic targets to be manufactured. And what we've seen now even going back to as early as April was that many manufacturing houses had been synthesizing these templates, which are then at very high concentration alongside the assay oligos, and there's become cross-contamination between those. So that means that you're seeing these products in the oligo so that the actual reagents are contaminated. Pia Abola: So let's say you're in a diagnostic lab and you've got a positive result. How do you decide if that's a false positive or not? And how can you manage when your reagents are contaminated? Tania Nolan: Really good question. Absolutely. And this is where it comes back to talking about the associated controls so the negative control. If the negative is also positive you know for a fact that that is a false positive, and that is enough to raise concerns about other positives that you've run alongside. Where you start to have problems is for example, when we talked about that transfer. If there is transfer of material from something which is genuinely positive to a sample, which is genuinely negative but becomes contaminated without any transfer into your negative control, I'm not really sure how you would detect that. That's why we need to talk so much about absolute caution. Pia Abola: And then I was curious about from the workflow perspective and automation perspective, are there things you can do during automation and analysis of the assay that can help minimize false positives? James Grayson: Where automation can help and where software can help is we can look at those curves independent of human review. We can pass that all through algorithms, artificial intelligence that can assess that curve in a way to verify the shape, the performance, the features of the curve and confirm whether there's something is false positive... Well, in this case, in this example of false positive. Or truly negative just by looking at the shape and interpretation, we can do that with software with automation, for in some cases, hundreds of thousands of samples today in my personal experience with COVID and the projects that I help support. And thinking about how much data can be passed can go through a lab that would need to be reviewed where you rely on automation, where you rely on softwares to help filter that out to really verify that this curve is positive, this curve is negative and try to weed through anything that is anomalous. And therefore try to make that first pass removal of anything that might lead to a false positive or false negative from a curb morphology point of view. Jim Huggett: A couple of things you can also do is the controls. One of the things because SARS-CoV-2 is an RNA virus, and most of the contamination that we're talking about is DNA. We can run RT controls where you don't actually include the step to do the reverse transcription. So consequently there should be no signal if it is an RNA molecule, whereas if a DNA molecule is present people still see a result. And it immediately tells you that you've got DNA contamination and the SARS-CoV-2 coronavirus has no DNA stage in its life cycle. And so that's quite a neat way of working in that context. Pia Abola: I wanted to follow up on that a little bit. So the queue in qPCR is for quantitative, does the fact that you have to go through the reverse transcription step because it's an RNA virus, SARS-CoV-2 is an RNA virus. Does that affect the quantitative nature of this assay? Jim Huggett: So yes, potentially there are many things that can influence that. And so the qPCR quantification cycle is quite a... it's a rough indicator. But for it to be properly quantitative, we need good levels of calibration and to understand what may be contributing to the sources of bias. Pia Abola: So the current SARS-CoV-2 assays that are on the market right now, those cannot be used for quantitative measurement of viral load say? Jim Huggett: Can they not be used? People are using them quantitatively. They are saying, people are trying to apply cutoffs. They're saying high late 30 cycles maybe we can ignore them. Where quantification is conducted robustly in clinical virology, is in things like HIV, hepatitis B, viral load testing, patients are monitored, their viral load is monitored. There's a rigorous system of infrastructure supporting this via the WHO reference materials and this is a systemic virus. You take a sample of plasma from the arm, which is what we've measured. Generally, it's not homogeneous but it's generally well spread throughout the plasma. In the case of SARS-CoV-2 it can be in a particular region it moves around. We know as the disease presents, it goes into the lungs, it may not be found in the oral cavity anymore. Jim Huggett: And so you have that immediate challenge is the fact that where you sampled may or may not contain virus. We know from large powered studies, that the viral burden will vary as well during the course of the disease. So it's how you want to use that quantitatively is an issue as well. And we know that the analytical issues where it's achieved and where it worked in HIV, for example, there's an entire infrastructure to support this that does not exist for SARS-CoV-2. Tania Nolan: And in terms of being quantitative, qPCR can only be quantitative if like most other analytical techniques you've got some kind of standard to measure against. Otherwise, we have relative quantities between samples which we can calculate. But in terms of being quantitative and measuring viral load, you need some kind of accurate standard that everybody agrees is quantified to a certain degree. Pia Abola: So you mentioned Jim that the infrastructure for quantitating SARS-CoV-2 is being developed. Are there any reagents that we have now or what's on the horizon for that? Jim Huggett: So there are a number of sources of quality control materials and proficiency testing, external quality assurance materials that vary from whole cultured virus, to RNAs that has been made into some kind of armored shell or some other virus that you can use and that tend to be more safe. That would allow you to control for the extraction versus the whole process. And then there are a number of materials, RNA materials whole viral extracts, in vitro transcriber, there's an entire array of different things that are available. And some of these are used to support... Well, they're certainly used to support proficiency, and they are the only things that can be used to report performance like limited detection, as is needed for certain accreditation Emergency Use Authorization in the US. But many of these materials are not necessarily designed for that. Jim Huggett: And the vendors have not... they're not selling them for that. But are the only things that are available at the moment, which give you quality. And that's one of the issues there's a juxtaposition between the virus... We're interested in whether the virus is present, what that means in the context of RNA, Tania alluded to we can talk about or not. But the presence of the virus is what's being measured is a patient being infected, using a quite precise, potentially quantitative method. And so you have this either other side of arguments with the limit of detection as a good example, or trying to cut thresholds where you can't really do that because of the variation we've talked about. And so this becomes an issue in that context. Pia Abola: James, what are the software solutions or automation solutions to help with proficiency testing. James Grayson: The kind of software solution you need to support that would be something akin to a business intelligence software. So you're looking at a higher level of data collection of data attraction, where you're pulling the individual results from given sites and aggregating them. And I think the most interesting thing that I've seen as far as practical implications of a proficiency program is how standardized that is. And so from a software perspective you can monitor that, you can look for these variations based off of what proficiency program is implemented. But I think there's a deeper question about is that proficiency program viable for a situation like COVID. But I think the bigger question is actually the interesting one and I don't have an answer to it. I just think it's really incredible to see where we are as a community, a global community trying to battle this disease. Pia Abola: Tania, do have any comments about proficiency testing and for quality control? Tania Nolan: I think James certainly described the problem. I think if you're going to have this kind of quality standardization, then ideally you've got very few standards that more people are referring to. If you're going to have as many standards or quality perspectives, then clearly we need to know how they all stand up against each other. If you go right back to the beginnings of qPCR and the MIQE paper from many years ago, we talked a lot there about defining the quality of individual assays within the context of using your own standards. But then we still have the challenge of how we map that standards data against what somebody else might be using. But even simple measures around ensuring that you have efficient assays, and knowing what the limits of detection of those assays is. It certainly goes some way to, for example, avoiding false negatives by using inefficient assays or insensitive assays. Pia Abola: Before we talk about limit of detection, I'm curious if any of you think that there are ways that the pandemic is confusing molecular diagnostic assay developers. In that we have all this experience with these endemic diseases and testing for endemic diseases, do we need to treat SARS-CoV-2 testing differently? Is it affecting how we think about SARS-CoV-2 testing? Tania Nolan: I think also maybe we didn't appreciate the scale of the pandemic, and possibly the scale of the testing regimes that would be helpful. And so maybe thinking a little bit beyond the normal testing schemes that we have and we're comfortable with and starting to think about alternatives. Like for example, pooling samples from households or pooling samples from whatever you class your bubble as. Even looking at the statistics around whether end point determinations are adequate to control or have an insight into the population incidents. So I think we went straight to full on real-time PCR because that certainly seems like the obvious choice but as we move forward, there are people looking at much more high throughput and potentially not just using the qPCR technologies. Pia Abola: I want to take a minute and talk about limit of detection and false negatives. So we talked about false positives. We talked about how false positives can arise from actual contamination. So something that can be amplified in the sample as well as instrument issues so drift in the signal. What about false negatives? Where are those coming from? How did those arise? Tania Nolan: Well, one of the things we talked about my little earlier was a poorly designed assay, poorly designed and put poorly optimized. And Jim was talking about the idea of thousands of fold difference in detection, just by how you do the analysis. We'll if you add on top of that an assay that's poorly designed, poorly optimized, potentially not well-made oligos then that shifts the detection of that same concentration of input to later cycles. And as you push that to later cycles clearly reached the point where you're not going to detect what is really a positive sample. So then you've got a genuine clinical negative, so a false negative in that case. Pia Abola: I'd like to turn this to James. You talked about during production to detect false positives, you can look at the neighboring wells. James Grayson: Oh, for contamination. Yeah. Pia Abola: For contamination. Can you use those same strategies for false negatives? James Grayson: It's a different situation. I think from a system or software perspective, software people like to use the term garbage in, garbage out. So if you don't have something viable going in, you're never going to get a result coming out. And I think when you look at false negatives, you can have instrument artifacts, you can have behavioral issues, but I think it is more workflow or systemic that that can lead to missing that with software. So if you consider the whole workflow, you've got issues of were the reagents dispensed correctly along the way. So was the extraction performed correctly, whether the primers and probes, the enzymes added if at all were in the correct amounts and the correct ratios. And so the lack of amplification there is obviously no amplification, you're going to interpret that as no signal or and a quote unquote negative result. But then we do have the checks of internal controls and played controls and played validity checks to try to weed those out. James Grayson: I think really from a more practical aspect though, it does come back to interpretation of those low late amplifications. And I think that's where I potentially see the most variation. If you did get amplification, if it was a signal, if it was quote unquote positive and is a positive sample did it just miss an arbitrary cutoff? And I think that's, once again, this is where we see a lot of variation in that interpretation variation of how the interpretation rules that'll come in the SOP or instructions for use from a kit. And what do we call it? And you could easily see how if somebody made a wrong interpretation, or set up a wrong rule because of the data set that they started with because of the sample set that they started with. James Grayson: And they drew a line at cycle 35 say, and you got an amplification that maybe is a genuine amplification that came up at 35.01 cycles. By rule that might be considered a negative and that's... but there's also the clinical relevance there. And that's something that I don't think we're entirely aware of yet either, what is the clinical relevance? What is the significance of a low late amplification? And I think that definitely feeds into the variation in interpretation. So from a software perspective it's as simple as that, do you set up rules? Do you have rules? Did you meet those rules? Did you get amplification? And I think the most variation and the most detectable form of false negatives are these low late amplifications and how people interpret them. James Grayson: And you get different opinions and it's hard to know what is the right way to go. So we have to trust local implementations. We have to trust that the clinician knows how to best serve their population in a situation where they're cobbling together their own solution. So it's very challenging I would say from any software system perspective to really nail that down. Pia Abola: Any other comments on that? Jim Huggett: On the limit of detection, what the SARS-CoV-2 pandemic and the fact that every reporter on the planet is almost interested in molecular diagnostics at the moment is really showing, is that the limited detection is a quantitative metric on what is essentially a yes, no test and it's really key. And if you look at the Emergency Use Authorization in the US there are over four orders of magnitude variation in the reported limited detection for these molecular tests. Now some of them are saying they can measure 10 copies/ml, which is a pretty astounding a feet and I... Yeah, anyway. But some are happily submitting over a 100,000 copies/ml. Now there may be arguments that you might find that you don't need less than a 100,000 copies/ml because it's a low level RNA, it's not relevant, it's not infectious. Jim Huggett: That may be where we go, but we don't know that yet. And it perplexes me how with the Emergency Use Authorization my understanding is that people are marking their own homework, and they're doing those experiments with some standard that they've gotten and they're running that. And so it's surprising that that spectrum of variation is seen as okay to submit. Now, there are reasons why you'd be talking about the assay designers, whatever, but a major reason for this will also be the upstream processes involved in taking the sample. You only take a hundred microliters there's only... One method takes a hundred microliters and the other takes a ml because of physics, the latter is going to be 10 fold higher in its potential limited detection. What it can measure per ml of virus, there are advocates and many potential methods where they're trying to speed things up with direct addition. Jim Huggett: You take the sample or the viral transport media swab, and you put that straight into the experiment. There's an isothermal method that'll allow you to do that. Well, they cannot measure, they cannot take advantage of the extractions concentration step, and that's nothing to do with the technology it's physics. And these are things that will contribute to that, how well that works. What is needed is a really good one, one of the things we may find out when we look back at this is that we went in with the most analytical sensitive technique that exists to measure this virus. And that may potentially have almost hindered how we respond to it because we're measuring everybody. Whereas if our method happens to be a hundred times less sensitive, we may have been in a much better position ironically, to pick up those patients who have more virus or maybe they are more infectious. Jim Huggett: This is not known, I add that bit. But I'm hypothesizing that we may find that if that turns out to be the case. And certainly if we're going to push for other methods, which we know are likely to be less sensitive such as antigen testing, we're going to have to probably accept to need to have that hit, that we're going to be measuring... We're not going to have the same level of analytical sensitivity to find the virus wherever it is, but that may be okay because that provides you with sufficient sensitivity from a clinical point of view. And it's perfectly fit for purpose in its use, in responding to this pathogen. Tania Nolan: If this virus is mutating then we need to stay on top of that as well, and recognize that if we've got a design that was done last March, does that design still detect all mutations of the virus, or is that something that we have to modify and account for as well. Pia Abola: And so how do you design an assay to do that? Is it frequent checks of sensitivity or? Tania Nolan: Usually, when you do the design you can if you're fortunate enough to have other organisms that you can compare to, then as those first sequences of SARS 2 were coming through and being published, we were in a position of being able to compare these to other coronaviruses and look for regions of consensus. And it's a reasonably safe assumption if the virus has consistent regions, then those are less likely to be mutated, they're clearly functional. So one of the ways that you will design will be to target some of those, of course, in that case there's a risk, again, depending on how you're doing the design. That you would potentially pick up people with either SARS 2 or the common cold, so clearly you need to ensure that your design is going to differentiate between those. Jim Huggett: What they do in my understanding is what they're doing in the clinical diagnostics with certainly for viruses is you measure more than one target. And so should you get one changing, you have the backup of the other one and being there. And also in the case of the pandemic we are experiencing at the moment, I don't think anything has been sequenced quite as extensively as SARS COVID. So this is constantly happening, constantly being monitored. And so it's very important that those who develop the methods and those who apply them, keep an eye on this to make sure that they're comfortable that their regions are not changing, genetically. Pia Abola: I was wondering if each of you could provide one most important piece of advice you would give to an assay developer, looking to minimize false positives or false negatives. Tania maybe we can start with you. Tania Nolan: Oh, controls. I'm a great advocate of controlling and adding more controls. So during the development stage absolutely ensuring that you have your standards, that you fully understand how those are performing. You understand your limit of detection and you have controls that are then carried forward into the analytical sphere. Pia Abola: Jim. Jim Huggett: So I would say when you're designing your assay, go back to the first principles, look at some of the earlier stuff, the MIQUE guidelines. But also in the case of the actual PCR itself, not the... So the limit of detection talks about per ml at the whole virus, and that experiment you were running, you should be able to detect near single molecules. Some of the reports you'll see some of the PCRs they're measuring 50, a 100 copies of RNA in an individual reaction. We really should be aiming to get it working as well as it can. Now, the reason for that is not just because of that limit of detection which may as I mentioned earlier on turn out to be not what we want, is because that is also a surrogate of how well the experiments working, how well it's performing at its most optimum. Jim Huggett: And so we really should be aiming for that, these should be the case that we do that. And I agree with Tania from the point of view of control. One additional note though with the controls for the false positives, is that we should think of carrier containing controls by which they have some kind of RNA like the real clinical sample. Because a water only or a buffer only no nucleic acid control is not the same as the real thing. And we know that if you have some carrier RNA or DNA or nucleic acid floating around, it can improve the sensitivity of your method. So that may potentially allow you to find very low level contamination that you don't spot in a non carrier containing RNA control, and therefore give you a false idea that these are not false positives. Pia Abola: And then James. James Grayson: This is going back to a previous life and a previous work experience. But I think for the developer, something that I've seen in working for an assay development company is demographics, representation. We talked about mutation, we talked about variation in the sequence, and I've worked on products that did not take that into account where the assay was designed using only clinical samples from Caucasians in the United States. And so therefore when you roll that product out, you can see vastly different performance, those products are fortunately no longer on the market. And that was from a well-oiled company, that was doing things as best they could and as appropriately as they could. And then once again you have... To me at least I have some level of concern in hundreds of kits on the market and how fast they're coming out, and how well they represent. James Grayson: Another thing from personal experience is in the development process, in the testing processes, is clinical samples versus contrived samples. You can see vastly different performance using samples from actual patients versus some spiked media. And then I think making sure that you have a good representation there is also critical to get the most accurate and viable result. Pia Abola: And then I just have one more question for any clinicians that might be listening. What do each of you think is the most important thing for them to know about the state of SARS-CoV-2 testing as of October 2020. Jim Huggett: We're not ready to quantify the virus yet. And at analytical reasons setting thresholds at this particular point in time based on quantification cycle is not really possible. Well, it's possible but it will not give you a robust quantitative measure across labs. We at the very least need to wait for the WHO reference material to become available before we can even begin to think to do that. Pia Abola: Tania. Tania Nolan: Trust your instincts. A lot of clinical knowledge and these tests are aides to your diagnosis. We know that without doubt in the early days there were very, very high rates of false negatives. And so it's important to balance all of the information. Pia Abola: And then James. James Grayson: I would say standardize. We've talked about all the variation points that exist right now in trying to monitor for COVID. And historically pre-pandemic a clinician might buy an assay that came from a provider and it was a fully contained system having all the upfront robotics, the detection system, the assay, everything is well defined. And now a clinician has to know their process from A to Z, they potentially have to know every component that's going into cause variation. And having to do that in a resource limited environment where maybe supply chain is failing, maybe the workforce is changing. Maybe you need to hire more people and they're less trained. So looking for tools in this case, automation tools, software intelligence, to help you standardize, to help you track, to help you produce the same result consistently. At least as a starting point to start looking for variation and really understanding your process and getting the best result for your population. Pia Abola: Excellent. Well, thank you everybody for the discussion it was an excellent discussion. Really appreciate your input. Thank you for joining us for this episode of Countering COVID. If you're interested in additional resources on this topic, you can check the show notes or visit us at biosearchtech.com/COVID-19. That's biosearchtech.com/COVID-19. Thank you.