Advertisement

Development and validation of an intraoperative bleeding severity scale for use in clinical studies of hemostatic agents

Open AccessPublished:November 10, 2016DOI:https://doi.org/10.1016/j.surg.2016.09.022

      Background

      Clinical studies investigating topical hemostatic agents have not used standardized definitions for intraoperative bleeding. The Food and Drug Administration has recently sought use of a validated, clinician-reported scale to standardized bleeding sites in these clinical studies. The intent of a scale is to reduce patient risk, generate labeling claims, and allow comparisons among study results. We describe the development and validation of an intraoperative bleeding severity scale.

      Methods

      A concept phase defined the framework of the scale. A feasibility and validation phase investigated the usability, clarity, relevance, and reliability (ie, intra- and interobserver concordance) among surgeons and surgical specialties as required by the Food and Drug Administration for the validation of a clinician-reported scale. Data were collected using an online tool. A total of 144 surgeons participated in the 3 phases.

      Results

      The scale developed during the concept phase achieved an average intraobserver concordance of 0.97 and an interobserver concordance of 0.89 in the feasibility phase (N = 33); a concordance of 1.0 is perfect. The scale was refined and then achieved an average intraobserver concordance of 0.98 and an interobserver concordance of 0.91 in the validation phase with unanimous agreement by surgeons from multiple surgical specialties that the scale can be implemented into clinical studies (N = 102).

      Conclusion

      This study validated an intraoperative bleeding severity scale for use in clinical studies investigating hemostatic agents. The scale was usable, clear, and clinically relevant with excellent reliability. The scale fulfills requirements of the Food and Drug Administration for a clinician-reported scale and can be used to generate clinically meaningful labeling claims.
      In the past century, surgical hemostasis has become a central tenet to the advancement of surgery.
      • Halsted W.A.S.
      The training of the surgeon.
      The use of hemostatic agents is a standard-of-care treatment in many surgical specialties
      American Society of Anesthesiologists Task Force on Perioperative Blood Management
      Practice guidelines for perioperative blood management: an updated report by the American Society of Anesthesiologists Task Force on Perioperative Blood Management.
      • Shander A.
      • Van Aken H.
      • Colomina M.J.
      • Gombotz H.
      • Hofmann A.
      • Krauspe R.
      • et al.
      Patient blood Management in Europe.
      • Ferraris V.A.
      • Brown J.R.
      • Despotis G.J.
      • Hammon J.W.
      • Reece T.B.
      • et al.
      Society of Thoracic Surgeons Blood Conservation Guideline Task Force
      2011 update to the Society of Thoracic Surgeons and the Society of Cardiovascular Anesthesiologists blood conservation clinical practice guidelines.
      • Menkis A.H.
      • Martin J.
      • Cheng D.C.
      • Fitzgerald D.C.
      • Freedman J.J.
      • Gao C.
      • et al.
      Drug, devices, technologies, and techniques for blood management in minimally invasive and conventional cardiothoracic surgery: a consensus statement from the International Society for Minimally Invasive Cardiothoracic Surgery (ISMICS) 2011.
      • Kommu S.S.
      • McArthur R.
      • Emara A.M.
      • Reddy U.D.
      • Anderson C.J.
      • Barber N.J.
      • et al.
      Current status of hemostatic agents and sealants in urologic surgical practice.
      with increased use in major surgeries.
      • Wright J.D.
      • Ananth C.V.
      • Lewin S.N.
      • Burke W.M.
      • Siddiq Z.
      • Neugut A.I.
      • et al.
      Patterns of use of hemostatic agents in patients undergoing major surgery.
      • Tollec S.M.
      • Trossaert M.
      • Duveau D.
      • Grimandi G.
      • Sellal K.O.
      A proposed model for evaluating surgical hemostatic agents.
      Inadequate or incomplete hemostasis significantly increases the risk of perioperative morbidity and mortality,
      • Murphy G.J.
      • Reeves B.C.
      • Rogers C.A.
      • Rizvi S.I.
      • Culliford L.
      • Angelini G.D.
      Increased mortality, postoperative morbidity and cost after red blood cell transfusion in patients having cardiac surgery.
      • Glance L.G.
      • Dick A.W.
      • Mukamel D.B.
      • Fleming F.J.
      • Zollo R.A.
      • Wissler R.
      • et al.
      Association between intraoperative blood transfusion and mortality and morbidity in patients undergoing noncardiac surgery.
      significantly increases health care costs and resource use,
      • Dimick J.B.
      • Chen S.L.
      • Taheri P.A.
      • Henderson W.G.
      • Khuri S.F.
      • Campbell Jr., D.A.
      Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program.
      • Shander A.
      Financial and clinical outcomes associated with surgical bleeding complications.
      • Christensen M.C.
      • Krapf S.
      • Kempel A.
      • von Heymann C.
      Costs of excessive postoperative hemorrhage in cardiac surgery.
      • Stokes M.E.
      • Ye X.
      • Shah M.
      • Mercaldi K.
      • Reynolds M.W.
      • Rupnow M.F.
      • et al.
      Impact of bleeding-related complications and/or blood product transfusions on hospital costs in inpatient surgical patients.
      and is worsened by the use of antiplatelet and antithrombotic agents.
      • Levy J.H.
      • Dutton R.P.
      • Hemphill 3rd, J.C.
      • Shander A.
      • Cooper D.
      • Paidas M.J.
      • et al.
      Multidisciplinary approach to the challenge of hemostasis.
      • Yamamoto K.
      • Wada H.
      • Sakakura K.
      • Ikeda N.
      • Yamada Y.
      • Katayama T.
      • et al.
      Cardiovascular and bleeding risk of non-cardiac surgery in patients on antiplatelet therapy.
      • Devereaux P.J.
      • Mrkobrada M.
      • Sessler D.I.
      • Leslie K.
      • Alonso-Coello P.
      • Kurz A.
      • et al.
      Aspirin in patients undergoing noncardiac surgery.
      The need for hemostasis has led to the development of several topical hemostatic agents.
      Clinical studies investigating topical hemostatic agents have not used standardized definitions or classifications for intraoperative bleeding severity or hemostasis,
      • Schuhmacher C.
      • Pratschke J.
      • Weiss S.
      • Schneeberger S.
      • Mihaljevic A.L.
      • Schirren R.
      • et al.
      Safety and effectiveness of a synthetic hemostatic patch for intraoperative soft tissue bleeding.
      • Fischer C.P.
      • Bochicchio G.
      • Shen J.
      • Patel B.
      • Batiller J.
      • Hart J.C.
      A prospective, randomized, controlled trial of the efficacy and safety of fibrin pad as an adjunct to control soft tissue bleeding during abdominal, retroperitoneal, pelvic, and thoracic surgery.
      in part due to the lack of consensus in defining bleeding severity and in part due to the lack of requirement to do so. As a result, the labeling claims of hemostatic agents lack differentiation (eg, the appropriate severity or type of bleeding to be treated) (Online Supplement: eTable I). The generic labeling claims further impede surgeons' selection of the appropriate agent, conceal meaningful clinical differences, and lead to inefficiencies in surgery.
      • Tollec S.M.
      • Trossaert M.
      • Duveau D.
      • Grimandi G.
      • Sellal K.O.
      A proposed model for evaluating surgical hemostatic agents.
      The use of a validated bleeding severity scale can establish standardized inclusion and exclusion criteria in a clinical study so that appropriate labeling claims can be generated. Furthermore, the use of standardized criteria ensures that patients are not subject to undue risks (eg, failed, delayed, or use of inappropriate treatments). With standardized criteria, patient and clinical study outcomes can also be compared to determine relative effectiveness.
      Given these benefits, the United States Food and Drug Administration (FDA) now requires the use of a validated bleeding severity scale in clinical studies investigating hemostatic agents. Since no intraoperative bleeding severity scale has been validated for open surgical procedures and is applicable to multiple surgical specialties, the objective of this study was to develop and validate a clinician-reported scale (CRS) for intraoperative bleeding severity for use in clinical studies of hemostatic agents.

      Methods

      The FDA provides guidance for the creation, development, and validation of a CRS.
      United States Food and Drug Administration
      Clinical outcome assessment qualification program [Internet].
      Within this guidance, there are specific criteria that must be fulfilled for acceptability and validation (Table I). These criteria were addressed throughout the development and validation in which surgeons from different surgical specialties developed and applied a scale to the full spectrum of bleeding as depicted in several videos.
      Table ICriteria for a clinician-reported scale (CRS) based on the Food and Drug Administration's essential elements and criteria to evaluate a CRS
      ElementCriteria definition
      Ability to detect changeClinicians' scores change in response to changes of the intended measurement concept across the entire range that is clinically expected.
      ClarityCRS is reported as clear by a large segment of the user population.
      Construct validityItems and domains of a CRS are appropriate and comprehensive relative to its intended measurement concept, population, and use.
      RelevanceCRS is reported as relevant by a large segment of the user population.
      RepeatabilityCRS use provides the ability to yield consistent, repeatable results from the same clinician (ie, intraobserver).
      ReproducibilityCRS use provides the ability to yield consistent, reproducible results from multiple clinicians (ie, interobserver).
      Response rangeClinicians use both ends of the scale, response choices apply, and distribution is not highly skewed.
      UsabilityClinicians are able to use the scale, as well as comprehend, retain, and accurately follow instructions.

      Bleeding videos

      Clinical videos lack standardized quality and the ability to predict the level of bleeding and measure the rate of blood loss; therefore, an animal model was required to create the videos. All animal activities were performed according to the Guide for the Care and Use of Laboratory Animals and the United States Animal Welfare Act in an institution accredited by the Association for Assessment and Accreditation of Laboratory Animal Care International following Institutional Animal Care and Use Committee approval.
      Pigs were selected due to their anatomical size and organ structure, which are similar to that of humans.
      • Swindle M.M.
      • Makin A.
      • Herron A.J.
      • Clubb Jr., F.J.
      • Frazier K.S.
      Swine as models in biomedical research and toxicology testing.
      Eight male, domestic pigs were used to create videos representing the full range of blood loss across multiple surgical specialties and coagulopathic conditions (Fig 1). Male pigs were used to avoid coagulation differences related to estrus. No adjunctive hemostatic agents were used in the videos (eg, oxidized cellulose, collagen, fibrin glue, etc).
      Figure thumbnail gr1
      Fig 1Bleeding severity models. Various depictions of clinically relevant bleeding performed in a porcine animal model; specifically, (A) aortotomy closure site, (B) untreated, superficial hepatic abrasion, (C) untreated, deep nephric abrasion, (D) untreated intramammary dissection with vascular rupture, (E) untreated hepatic laceration, (F) untreated, superficial nephric abrasion, (G) untreated partial cystotectomy, (H) superficial, cardiac laceration, (I) untreated partial hepatectomy, and (J) untreated, supra-renal abdominal aortotomy.
      For each bleeding model, the rate of blood loss was measured by collecting blood from each lesion with preweighed gauze for 6 seconds then calculating the difference and multiplying by 10 to obtain a rate in milliliters per minute, where 1 g of blood loss equated to 1 mL.
      Videos were then selected based on representative rates of intraoperative blood loss, suitable cinematic quality, and clinical relevance (Online Supplement: eVideos 1–5). Each video was edited to be 15 seconds.

      Concept, feasibility, and validation study phases

      The development and validation of the scale had 3 study phases: concept, feasibility, and validation. During each phase, surgeons were recruited based on the anatomic cavity in which he or she performs surgery according to American Medical Association surgical specialties. Surgeons were identified based on demonstrated involvement or interest in clinical studies of hemostatic agents. Surgeons were not selected based on previous consultancy agreements, product usage, or industry affiliations. Once recruited, surgeons were only allowed to participate in one of the phases to avoid participant overlap and creation bias. Three surgeons from the concept phase, however, were selected as advisors and participated throughout the validation (P.E.S., E.L., A.D.). All surgeons were recruited by BioMedCom Partners, Inc (New York, NY).

      Concept study phase

      This phase defined the conceptual framework and the intended use of the scale. A minimum of 3 surgeons from the 3 surgical cavities, 9 surgeons total, were recruited to participate. Each surgeon was board certified and a recognized leader within their field of surgery. This study phase was performed face-to-face, during which qualitative data were collected.

      Feasibility study phase

      This phase investigated the construct validity, ability to detect change, clarity, relevance, and response range of the scale developed during the concept phase and determined the videos needed for the validation phase. A minimum of 10 surgeons from each of the 3 surgical cavities, 30 surgeons total, were recruited to participate. Each surgeon was board certified and had a minimum of 5 years of postgraduate surgical experience. This study phase was performed face-to-face, during which quantitative and qualitative data were collected via an online data collection tool. The online data collection tool was designed and implemented by BioMedCom Partners, Inc.
      Surgeons were didactically trained on the use of the scale and provided videos depicting different severities of bleeding specific to their own surgical cavity. Participants then individually used the scale to assess 15 videos relevant to their surgical cavity, after which they graded 15 additional videos from each of the other 2 surgical cavities. Each set of 15 videos contained 10 unique videos, of which 5 were repeated as duplicate videos. The 10 unique videos were used to assess interobserver agreement. The 5 duplicate videos were used to assess intraobserver agreement.

      Validation study phase

      This phase obtained statistically meaningful inter- and intraobserver agreement data using the scale and additional observational data supporting the scale's usability, clarity, and relevance. A minimum of 30 surgeons from each of the 3 surgical cavities, 90 surgeons total, were recruited to participate. Each surgeon was board certified and had a minimum of 5 years of postgraduate surgical experience. This study phase was performed entirely through the online data collection tool used in the feasibility phase, in which quantitative and qualitative data were collected.
      Surgeons were self-trained on the scale through visual presentation and training videos of each level of severity. Surgeons then used the scale to grade 17 videos not specifically related to their surgical cavity. Of the 17 videos, the first 2 videos were not analyzed, the next 10 videos were unique, and the last 5 videos were repeats. The first 2 videos allowed familiarization with the online tool, video and question format, and data entry. The 10 unique videos were to assess interobserver agreement. The 5 duplicate videos were to assess intraobserver agreement.

      Statistical analysis

      Agreement among surgeons and among surgical cavities (ie, interobserver agreement), and agreement within a surgeon and within a surgical cavity (ie, intraobserver agreement) was investigated. Interobserver agreement measures reproducibility, and intraobserver agreement measures repeatability of using a scale. Inter- and intraobserver agreement was analyzed using the Kendall's coefficient of concordance (Kendall's W) statistic. A Kendall's W of ≥0.7 is considered as “acceptable,” ≥0.8 as “appreciable,” ≥0.9 as “excellent,” and 1.0 as “perfect” concordance.
      • Lesaffre A.R.
      Statistical and methodological aspects of oral health research.
      • Schmidt R.C.
      Managing Delphi surveys using nonparametric statistical techniques.
      Box plots were used to present intraobserver Kendall's W statistics. Qualitative data were summarized using frequencies and percentages.
      The sample size of the validation phase was based on an experimental power calculation after the feasibility phase, in which a sample size of 90 subjects assessing 10 videos achieves >80% power to detect a Kendall's W of 0.80 for interobserver agreement, and assessing 5 videos twice achieves >80% power to detect a Kendall's W of 0.80 for intraobserver agreement.

      Results

      Concept study phase

      Nine surgeons participated (Table II). The average years of practice was 15 (standard deviation = 6.0). All surgeons reported using hemostatic agents in practice. The surgeons critically assessed the possible inclusion items and domains for a scale, which were then narrowed to those deemed most relevant (ie, anatomical appearance, intervention, qualitative description, visual appearance, and visually estimated rate of blood loss). The surgeons then defined the items and domains according to a quantitative structure, where grade 0 was “no bleeding” and grade 4 was “life-threatening bleeding.”
      Table IISurgeon participants
      Concept (N = 9)Feasibility (N = 33)Validation (N = 102)
      Per surgical specialty
      Neurological, Ophthalmic, Oral and maxillofacial, Orthopedic, Otolaryngology, Pediatric, Plastic, and Maxillofacial specialties were excluded.
       Bariatric3 (9.1)1 (1.0)
       Cardiac3 (33.3)10 (30.3)24 (23.5)
       Colorectal1 (11.1)1 (3.0)2 (2.0)
       General1 (11.1)3 (9.1)18 (17.7)
       Gynecologic oncology1 (3.0)8 (7.8)
       Hepatobiliary1 (11.1)4 (12.1)2 (2.0)
       Obstetrics and gynecology8 (7.8)
       Pulmonary
       Thoracic3 (2.9)
       Trauma1 (3.0)13 (12.8)
       Urology3 (33.3)9 (27.3)16 (15.7)
       Vascular1 (3.0)7 (6.9)
      Hospital setting
       Academic7 (77.8)26 (78.8)65 (63.7)
       Academic/nonprofit1 (11.1)5 (4.9)
       Academic/private1 (1.0)
       Academic/nonprofit/private3 (2.9)
       Nonprofit2 (6.1)7 (6.9)
       Private1 (11.1)4 (12.1)19 (18.6)
       Private/nonprofit1 (3.0)2 (2.0)
      Surgeon sample size and backgrounds per study phase. Surgeon involvement was essential for scale acceptability, credibility, clarity, relevance, and usability. Data are presented as number (percent).
      Neurological, Ophthalmic, Oral and maxillofacial, Orthopedic, Otolaryngology, Pediatric, Plastic, and Maxillofacial specialties were excluded.

      Feasibility study phase

      Thirty-three surgeons participated (Table II). The average years of practice was 16.6 (7.6). All surgeons reported having used hemostatic agents.
      All surgeons deemed a validated scale necessary for clinical studies, and 87.9% (n = 29) deemed it necessary for making labeling claims. Thirty-two surgeons (97.0%) agreed that the scale can be implemented for use in a clinical study. Thirty surgeons agreed that the scale was either “very” (n = 5, 15.2%) or “mostly” (n = 25, 75.8%) self-explanatory, while 3 (9.1%) surgeons held that it was “not really,” and none felt that it was “not at all” self-explanatory. All surgeons agreed that the scale reflected the range of bleeding site sizes and severities expected in clinical practice. Surgeon scores detected change across the videos, used the full response range, and responses changed appropriately to changes in severity (Table III).
      Table IIIResponse range and frequency of bleeding scores
      Bleeding severity gradeAbdominal cavity videosPelvic cavity videosThoracic cavity videos
      Video 1, bleed rate22.8 mL/min5.1 mL/min19.7 mL/min
       00 (0)0 (0)1 (3.0)
       11 (3.0)19 (57.6)1 (3.0)
       221 (63.6)12 (36.4)3 (9.1)
       310 (30.3)2 (6.1)25 (75.8)
       41 (3.0)0 (0)3 (9.1)
      Video 2, bleed rate0.6 mL/min16.5 mL/min14.6 mL/min
       011 (33.3)0 (0)0 (0)
       121 (63.7)0 (0)1 (3.0)
       21 (3.0)13 (39.4)12 (36.4)
       30 (0)20 (60.6)17 (51.5)
       40 (0)0 (0)3 (9.1)
      Video 3, bleed rate6.4 mL/min0.7 mL/min1.6 mL/min
       00 (0)32 (97.0)1 (3.0)
       11 (3.0)1 (3.0)29 (87.9)
       227 (81.8)0 (0)3 (9.1)
       35 (15.2)0 (0)0 (0)
       40 (0)0 (0)0 (0)
      Video 4, bleed rate0.5 mL/min366.7 mL/min176.3 mL/min
       022 (66.7)0 (0)0 (0)
       110 (30.3)0 (0)0 (0)
       21 (3.0)0 (0)1 (3.0)
       30 (0)1 (3.0)12 (36.4)
       40 (0)32 (97.0)20 (60.6)
      Video 5, bleed rate1.3 mL/min2.9 mL/min0.2 mL/min
       00 (0)0 (0)29 (87.9)
       17 (21.2)12 (36.4)4 (12.1)
       224 (72.7)21 (63.6)0 (0)
       32 (6.1)0 (0)0 (0)
       40 (0)0 (0)0 (0)
      Video 6, bleed rate5.5 mL/min15.2 mL/min9.7 mL/min
       00 (0)0 (0)0 (0)
       13 (9.1)0 (0)0 (0)
       226 (78.8)17 (51.5)1 (3.0)
       34 (12.1)16 (48.5)26 (78.8)
       40 (0)0 (0)6 (18.2)
      Video 7, bleed rate2.7 mL/min0.2 mL/min4.0 mL/min
       00 (0)29 (87.9)0 (0)
       123 (69.7)4 (12.1)14 (42.4)
       210 (30.3)0 (0)17 (51.5)
       30 (0)0 (0)2 (6.1)
       40 (0)0 (0)0 (0)
      Video 8, bleed rate366.7 mL/min46.8 mL/min40.4 mL/min
       00 (0)0 (0)0 (0)
       10 (0)0 (0)0 (0)
       20 (0)1 (3.0)1 (3.0)
       30 (0)22 (66.7)17 (51.5)
       433 (100.0)10 (30.3)15 (45.5)
      Video 9, bleed rate26.0 mL/min9.5 mL/min5.4 mL/min
       00 (0)0 (0)0 (0)
       11 (3.0)0 (0)3 (9.1)
       215 (45.5)20 (60.6)28 (84.9)
       317 (51.5)13 (39.4)2 (6.1)
       40 (0)0 (0)0 (0)
      Video 10, bleed rate84.8 mL/min4.5 mL/min0.1 mL/min
       00 (0)0 (0)5 (15.2)
       10 (0)5 (15.2)28 (84.9)
       21 (3.0)26 (78.8)0 (0)
       323 (69.7)2 (6.1)0 (0)
       49 (27.3)0 (0)0 (0)
      The entire response range was used by 33 surgeons assessing 10 unique videos from 3 different surgical cavities. Surgeon responses changed appropriately with changes in the bleeding severity demonstrating that the scale can detect change. The listed video order represents the order the videos were presented to surgeons. Data are listed as number of respondents (percent).
      The average intraobserver agreement for all surgeons was 0.97 for all types of videos, constituting “excellent” concordance (Table IV). All Kendall's W values were >0.7 for each surgeon when considering all surgical cavity videos (Fig 2). The overall interobserver agreement among all surgeons was 0.89 for all types of videos, constituting “appreciable” concordance (Table IV).
      Table IVFeasibility and validation concordance
      Kendall's coefficient of concordance (Kendall's W)Concordance status
      Feasibility phase average intraobserver agreement
       All surgeons, all videos (NSurgeons = 33, NVideos = 15)0.97Excellent
       Abdominal surgeons, all videos (NSurgeons = 12, NVideos = 15)0.97Excellent
       Pelvic surgeons, all videos (NSurgeons = 11, NVideos = 15)0.97Excellent
       Thoracic surgeons, all videos (NSurgeons = 10, NVideos = 15)0.95Excellent
       All surgeons, abdominal videos (NSurgeons = 33, NVideos = 5)0.98Excellent
       All surgeons, pelvic videos (NSurgeons = 33, NVideos = 5)0.97Excellent
       All surgeons, thoracic videos (NSurgeons = 33, NVideos = 5)0.94Excellent
      Feasibility phase interobserver agreement
       All surgeons, all videos (NSurgeons = 33, NVideos = 30)0.89Appreciable
       Abdominal surgeons, all videos (NSurgeons = 12, NVideos = 30)0.89Appreciable
       Pelvic surgeons, all videos (NSurgeons = 11, NVideos = 30)0.91Excellent
       Thoracic surgeons, all videos (NSurgeons = 10, NVideos = 30)0.90Excellent
       All surgeons, abdominal videos (NSurgeons = 33, NVideos = 10)0.88Appreciable
       All surgeons, pelvic videos (NSurgeons = 33, NVideos = 10)0.90Excellent
       All surgeons, thoracic videos (NSurgeons = 33, NVideos = 10)0.88Appreciable
      Validation phase average intraobserver agreement
       All surgeons, all videos (NSurgeons = 102, NVideos = 5)0.98Excellent
       Abdominal surgeons, all videos (NSurgeons = 38, NVideos = 5)0.98Excellent
       Pelvic surgeons, all videos (NSurgeons = 32, NVideos = 5)0.98Excellent
       Thoracic surgeons, all videos (NSurgeons = 32, NVideos = 5)0.98Excellent
      Validation phase interobserver agreement
       All surgeons, all videos (NSurgeons = 102, NVideos = 10)0.91Excellent
       Abdominal surgeons, all videos (NSurgeons = 38, NVideos = 10)0.93Excellent
       Pelvic surgeons, all videos (NSurgeons = 32, NVideos = 10)0.92Excellent
       Thoracic surgeons, all videos (NSurgeons = 32, NVideos = 10)0.88Appreciable
      In the feasibility phase, use of the Bleeding Severity Scale by 33 surgeons achieved “Excellent” intraobserver agreement and “Appreciable” interobserver agreement. Surgeons assessed 10 unique videos per surgical cavity to determine interobserver agreement and 5 videos twice per surgical cavity to determine intraobserver agreement. In the validation phase, 102 surgeons achieved “Excellent” concordance status for intraobserver and interobserver agreement. Surgeons assessed 10 unique videos to determine interobserver agreement and 5 videos twice to determine intraobserver agreement. A Kendall's coefficient of concordance (Kendall's W) of 1.0 is “Perfect” agreement.
      Figure thumbnail gr2
      Fig 2Intraobserver agreement values in the feasibility and validation phase. In the feasibility (top pane) and validation (bottom pane) phases, all surgeons, collectively and per specialty, achieved a Kendall's coefficient of correlation (Kendall's W) > 0.7, indicating that the bleeding severity scale was reliable. No outliers were observed in any of the groups.
      Surgeons of different surgical specialties repeatedly and reproducibly assessed bleeding severities of their own surgical cavity and of other surgical cavities. For this reason, videos specific to each surgical cavity were not required for the validation phase. The item titled “Intervention” introduced confusion and differed among the specialties, so it was removed for the validation phase (Table V).
      Table VValidated bleeding severity scale
      GradeVisual presentationAnatomic appearanceQualitative descriptionVisually estimated rate of blood loss (mL/min)
      0No bleedingNo bleedingNo bleeding≤1.0
      1Ooze or intermittent flowCapillary-like bleedingMild>1.0–5.0
      2Continuous flowVenule and arteriolar-like bleedingModerate>5.0–10.0
      3Controllable spurting and/or overwhelming flowNoncentral venous- and arterial-like bleedingSevere>10.0–50.0
      4Unidentified or inaccessible spurting or gushCentral arterial- or venous-like bleedingLife threatening
      Systemic resuscitation is required (eg, volume expanders, vasopressors, blood products, etc).
      >50.0
      The scale is designed and validated for use in clinical studies to generate labeling claims. The scale is a Likert-type scale, in which the user assigns a grade based on the overall agreement of the items listed.
      Systemic resuscitation is required (eg, volume expanders, vasopressors, blood products, etc).

      Validation study phase

      A total of 102 surgeons participated (Table II). The average years of practice was 14.7 (7.6). All surgeons but one reported having used hemostatic agents.
      All surgeons agreed that the scale can be implemented for use in a clinical study investigating the efficacy of a hemostatic agent. All surgeons agreed that the scale was either “very” (n = 53, 52.0%) or “mostly” (n = 49, 48.0%) self-explanatory, while none held that it was “not really” or “not at all” self-explanatory. All surgeons but one (n = 101, 99.0%) stated that the scale was relevant for evaluating hemostasis in clinical studies. Ninety-six surgeons (94.1%) agreed that the scale reflected the range of bleeding site sizes and severities expected in clinical practice, while 6 (5.9%) did not. Ninety surgeons (88.2%) agreed that the scale uses objective terms, while the remaining 12 surgeons (11.8%) agreed that the scale “does not but it does not prevent use.”
      Similarly, 82 surgeons (80.4%) agreed that the scale uses nonoverlapping terms, while the remaining 20 surgeons (19.6%) agreed that the scale “does not but it does not prevent use.” For surgeons who perform minimally invasive surgery, 79 of 99 (79.8%) stated that the scale can be applied “as is,” while the remaining 20 (19.6%) stated that it can be applied “with modification.” Of the 12 videos presented, 8 were determined to depict a clinical bleed by 92 or more surgeons (≥90.2%), 1 video by 63 surgeons (61.8%), and the remaining 2 videos were not determined to depict a clinical bleed. These 2 videos were assessed as a grade 0 (“No bleeding”), so the videos were not considered as depicting a bleed by the majority of surgeons.
      The intraobserver agreement among all surgeons was 0.98, constituting “excellent” concordance (Table IV). All Kendall's W values were >0.7 for each surgeon (Fig 2). The overall interobserver agreement among all surgeons was 0.91, constituting “excellent” concordance (Table IV).

      Discussion

      We describe the concept, development, and validation of a bleeding severity scale for use in clinical studies to generate labeling claims. The scale fulfilled all essential criteria required for a CRS with unanimous agreement to implement it into clinical studies. The usability, clarity, and relevance of the scale demonstrate the construct validity of the scale. And, the intra- and interobserver agreement demonstrate the reliability (ie, repeatability and reproducibility) of the scale. As a result, the scale is a validated means to establish labeling claims, standardize inclusion and exclusion criteria, and evaluate bleeding severity in open surgical procedures consistently across all surgical specialties in clinical studies. This scale addresses the FDA's request for a bleeding severity CRS that is also deemed necessary by surgeons.
      In prospective clinical studies investigating safety and effectiveness of hemostatic agents, bleeding severity is not defined consistently, and different levels of bleeding are deemed appropriate for the same control agent.
      • Schuhmacher C.
      • Pratschke J.
      • Weiss S.
      • Schneeberger S.
      • Mihaljevic A.L.
      • Schirren R.
      • et al.
      Safety and effectiveness of a synthetic hemostatic patch for intraoperative soft tissue bleeding.
      • Fischer C.P.
      • Bochicchio G.
      • Shen J.
      • Patel B.
      • Batiller J.
      • Hart J.C.
      A prospective, randomized, controlled trial of the efficacy and safety of fibrin pad as an adjunct to control soft tissue bleeding during abdominal, retroperitoneal, pelvic, and thoracic surgery.
      The definitions used in these studies and the chosen acceptable level of bleeding for the control agent are not known to be validated. As a result, the studies are not comparable and the results cannot be compared due to the different use of the same control agent. The use of the developed scale can be used to establish inclusion and exclusion criteria appropriate for a control agent to standardize study designs. Doing so reduces patient risk of experiencing intraoperative blood loss or postoperative complications due to hemostatic agent failure.
      In establishing the inclusion and exclusion criteria, a clinical study can target a specific bleeding severity level(s) to generate labeling claims specific to that level of bleeding severity. Alternatively, a scale can set a defined level of expected hemostatic performance, such as was developed for intraoperative surgical staplers.
      • Siegel J.M.
      • Cummings J.F.
      • Clymer J.W.
      Reproducible, repeatable and clinically-relevant hemostasis scoring.
      The scale developed by Siegel et al
      • Siegel J.M.
      • Cummings J.F.
      • Clymer J.W.
      Reproducible, repeatable and clinically-relevant hemostasis scoring.
      to evaluate the efficacy of surgical staplers achieved a Kendall's W for intraobserver agreement of 0.95 and interobserver agreement of 0.85 based on 154 surgeons. The scale has since been used to evaluate new surgical staplers.
      • Ng C.S.
      • Pickens A.
      • Siegel J.M.
      • Clymer J.W.
      • Cummings J.F.
      A novel narrow profile articulating powered vascular stapler provides superior access and haemostasis equivalent to conventional devices.
      While Siegel et al
      • Siegel J.M.
      • Cummings J.F.
      • Clymer J.W.
      Reproducible, repeatable and clinically-relevant hemostasis scoring.
      demonstrated reliability of their scale, they did not report on the validity of their scale (eg, construct validity, clarity, usability).
      In addition to the utility of a validated bleeding severity scale for use in human clinical studies, the scale can be used to standardize preclinical animal models for the development and comparison of hemostatic agents. The 2 bleeding severity scales used in animal models are not validated, though they are considered to be clinically relevant in scientific literature.
      • Jackman S.V.
      • Cadeddu J.A.
      • Chen R.N.
      • Micali S.
      • Bishoff J.T.
      • Lee B.R.
      • et al.
      Utility of the harmonic scalpel for laparoscopic partial nephrectomy.
      • Adams G.L.
      • Manson R.J.
      • Hasselblad V.
      • Shaw L.K.
      • Lawson J.H.
      Acute in-vivo evaluation of bleeding with Gelfoam plus saline and Gelfoam plus human thrombin using a liver square lesion model in swine.
      Jackman et al
      • Jackman S.V.
      • Cadeddu J.A.
      • Chen R.N.
      • Micali S.
      • Bishoff J.T.
      • Lee B.R.
      • et al.
      Utility of the harmonic scalpel for laparoscopic partial nephrectomy.
      designed a 5-point, Likert-type scale to evaluate the efficacy of harmonic scalpels, where 0 is “no hemostasis” and 4 is “dry.” The scale's applicability is limited to the efficacy of harmonic scalpels. Adams et al,
      • Adams G.L.
      • Manson R.J.
      • Hasselblad V.
      • Shaw L.K.
      • Lawson J.H.
      Acute in-vivo evaluation of bleeding with Gelfoam plus saline and Gelfoam plus human thrombin using a liver square lesion model in swine.
      however, designed a scale to evaluate the efficacy of hemostatic agents. The scale is a 6-point, pictorial scale of bleeding severity, where 0 is “no bleeding” and 4 is “severe.” The scale was refined to remove a “0.5” severity level, which increased its use and acceptance.
      • Lewis K.M.
      • Atlee H.D.
      • Mannone A.J.
      • Dwyer J.
      • Lin L.
      • Goppelt A.
      • et al.
      Comparison of two gelatin and thrombin combination hemostats in a porcine liver abrasion model.
      • Lewis K.M.
      • Atlee H.
      • Mannone A.
      • Lin L.
      • Goppelt A.
      Efficacy of hemostatic matrix and microporous polysaccharide hemospheres.
      • Lewis K.M.
      • Spazierer D.
      • Urban M.D.
      • Lin L.
      • Redl H.
      • Goppelt A.
      Comparison of regenerated and non-regenerated oxidized cellulose hemostatic agents.
      • Lewis K.M.
      • McKee J.
      • Schiviz A.
      • Bauer A.
      • Wolfsegger M.
      • Goppelt A.
      Randomized, controlled comparison of advanced hemostatic pads in hepatic surgical models.
      This preclinical scale has limited applicability beyond the lesion type described in these studies. Outside the use of the validated scale in research to standardize animal models, the scale can also provide context for residents and junior faculty who are learning hemostatic agent algorithms and surgeons who are unclear on when to use which hemostatic agent.

      Strengths

      A development process continually involving surgeons and a validation process involving surgeons from multiple surgical specialties are clear strengths. Interestingly, surgeons do not intuitively understand, interpret, and use CRSs consistently. The intra- and interobserver agreement of the scale developed in the concept phase increased from the feasibility to the validation phase after changes to the scale. The changes increased the usability of the scale leading to unanimous agreement that it can be implemented into a clinical study. In addition, the scale fulfilled all FDA criteria for a CRS.
      Surgeon assessments detected change appropriately with changes in the known bleeding rate and used the entire response range. The scale's reliability was seen within and across surgical specialties. Reliability was obtained from assessing a multitude of surgical procedural videos and coagulopathic conditions. A vast majority of surgeons agreed that the scale was relevant for evaluating hemostasis and that the range of bleeding site sizes and severities were appropriately represented.

      Limitations

      The validation of the scale was online and used videos depicting various bleeds, so use in a clinical setting can further support the scale's usability. Based on observational data in the validation phase, it is foreseeable that the scale will have high usability in the clinical setting. This is supported by all surgeons determining that the scale can be implemented for a clinical study and 99.0% of the surgeons determining that the scale is clinically relevant for evaluating hemostasis. The usability in the clinical setting may differ if the videos in the validation were not clinically relevant; however, all videos depicting an active bleed were considered to be clinically relevant by surgeons in the validation phase.
      Visual estimations of blood loss are known to be inaccurate.
      • Delilkan A.E.
      Comparison of subjective estimates by surgeons and anaesthetists of operative blood loss.
      • Schorn M.N.
      Measurement of blood loss: review of the literature.
      • Guinn N.R.
      • Broomer B.W.
      • White W.
      • Richardson W.
      • Hill S.E.
      Comparison of visually estimated blood loss with direct hemoglobin measurement in multilevel spine surgery.
      • Adkins A.R.
      • Lee D.
      • Woody D.J.
      • White Jr., W.A.
      Accuracy of blood loss estimations among anesthesia providers.
      • Dildy 3rd, G.A.
      • Paine A.R.
      • George N.C.
      • Velasco C.
      Estimating blood loss: can teaching significantly improve visual estimation?.
      As seen in our study and in the literature, estimations improve after training and when volumes are generalized to a range.
      • Dildy 3rd, G.A.
      • Paine A.R.
      • George N.C.
      • Velasco C.
      Estimating blood loss: can teaching significantly improve visual estimation?.
      Alternate techniques to estimate blood loss from a bleeding site include gravimetric, colormetric, photometry, spectrophotometry, and feature extraction imaging.
      • Schorn M.N.
      Measurement of blood loss: review of the literature.
      • Lamhaut L.
      • Apriotesei R.
      • Combes X.
      • Lejay M.
      • Carli P.
      • Vivien B.
      Comparison of the accuracy of noninvasive hemoglobin monitoring by spectrophotometry (SpHb) and HemoCue® with automated laboratory hemoglobin measurement.
      • Holmes A.A.
      • Konig G.
      • Ting V.
      • Philip B.
      • Puzio T.
      • Satish S.
      • et al.
      Clinical evaluation of a novel system for monitoring surgical hemoglobin loss.
      These traditional techniques have not proven valuable in the clinical setting due to the need for special equipment, the availability of staff, or the accuracy of the technique.
      Feature extraction technology, however, is a recent advancement that measures hemoglobin loss in real time with a high degree of accuracy.
      • Holmes A.A.
      • Konig G.
      • Ting V.
      • Philip B.
      • Puzio T.
      • Satish S.
      • et al.
      Clinical evaluation of a novel system for monitoring surgical hemoglobin loss.
      • Konig G.
      • Holmes A.A.
      • Garcia R.
      • Mendoza J.M.
      • Javidroozi M.
      • Satish S.
      • et al.
      In vitro evaluation of a novel system for monitoring surgical hemoglobin loss.
      • Sharareh B.
      • Woolwine S.
      • Satish S.
      • Abraham P.
      • Schwarzkopf R.
      Real time intraoperative monitoring of blood loss with a novel tablet application.
      Hemoglobin loss, however, does not represent blood volume loss, which determines bleeding severity. The applicability of this technology is, therefore, limited in its use for determining bleeding severity.

      Future research

      In our study, intraobserver agreement was assessed using a short, test-retest period. The intraobserver agreement may worsen with a longer test-retest period. The reliability of the interobserver agreement in the feasibility and validation phases, however, demonstrates that surgeons use the scale in a consistent way. This suggests that the intraobserver agreement is not likely to be affected over long periods of time.
      The surgical specialties selected to participate in this study are those that experience all levels of bleeding and are typically included in clinical studies investigating hemostatic agents. Based on the consistent agreement throughout all study phases among the included surgical specialties, the scale is likely to be broadly applicable to all surgical specialties. Data from the surgical specialties not included in the development and validation, however, would be further supportive of the scale's broad applicability.
      While a majority of surgeons foresee that the scale is applicable to minimally invasive surgery “as is,” a follow-up study is warranted to confirm reliability given the different environment, visibility (eg, magnification of bleeding, perception of bleeding), and physiologic effects (eg, hypertension secondary to insufflation, hypothermia secondary to nonhumidified or nonwarmed insufflation gases, decreased venous return, etc).
      Further, minimally invasive surgery has a lower tolerance for blood loss. This may suggest that the rates of blood loss may need to be revised. Based on the scale of Siegel et al
      • Siegel J.M.
      • Cummings J.F.
      • Clymer J.W.
      Reproducible, repeatable and clinically-relevant hemostasis scoring.
      for laparoscopic staplers, surgeons defined hemostasis as an oozing bleed that progresses for 15 seconds but does not require intervention. In comparison, this definition aligns to grade 0 “no bleeding” with a rate of blood loss of <1.0 mL/min. This suggests that the visually estimated rates of blood loss are translatable to minimally invasive surgery.
      In conclusion, we document the development and validation of an intraoperative bleeding severity scale that fulfills FDA criteria for a CRS. This validated scale is an essential tool for use in clinical studies to standardize inclusion-exclusion criteria and to assess the performance of hemostatic agents in clinical studies of hemostatic agents. The scale is designed to generate labeling claims that are clinically relevant, which can improve surgical efficiency by selecting the right hemostatic agent for the right bleed, ultimately reducing intraoperative and postoperative bleeding complications.
      The authors thank Drs John Olsen and Tage Carlson for their scientific input; Ellen Quinn, John Clay, Gina Gorman, and Milagrin Dsouza for their administrative support; Jesse Walker for video production; and the numerous surgeons and colleagues involved in the conception, development and validation of the bleeding severity scale.

      Supplementary data

      References

        • Halsted W.A.S.
        The training of the surgeon.
        Bull Johns Hopkins Hosp. 1904; 15: 267-275
        • American Society of Anesthesiologists Task Force on Perioperative Blood Management
        Practice guidelines for perioperative blood management: an updated report by the American Society of Anesthesiologists Task Force on Perioperative Blood Management.
        Anesthesiology. 2015; 122: 241-275
        • Shander A.
        • Van Aken H.
        • Colomina M.J.
        • Gombotz H.
        • Hofmann A.
        • Krauspe R.
        • et al.
        Patient blood Management in Europe.
        Br J Anaesth. 2012; 109: 55-68
        • Ferraris V.A.
        • Brown J.R.
        • Despotis G.J.
        • Hammon J.W.
        • Reece T.B.
        • et al.
        • Society of Thoracic Surgeons Blood Conservation Guideline Task Force
        2011 update to the Society of Thoracic Surgeons and the Society of Cardiovascular Anesthesiologists blood conservation clinical practice guidelines.
        Ann Thorac Surg. 2011; 91: 944-982
        • Menkis A.H.
        • Martin J.
        • Cheng D.C.
        • Fitzgerald D.C.
        • Freedman J.J.
        • Gao C.
        • et al.
        Drug, devices, technologies, and techniques for blood management in minimally invasive and conventional cardiothoracic surgery: a consensus statement from the International Society for Minimally Invasive Cardiothoracic Surgery (ISMICS) 2011.
        Innovations (Phila). 2012; 7: 229-241
        • Kommu S.S.
        • McArthur R.
        • Emara A.M.
        • Reddy U.D.
        • Anderson C.J.
        • Barber N.J.
        • et al.
        Current status of hemostatic agents and sealants in urologic surgical practice.
        Rev Urol. 2015; 17: 150-159
        • Wright J.D.
        • Ananth C.V.
        • Lewin S.N.
        • Burke W.M.
        • Siddiq Z.
        • Neugut A.I.
        • et al.
        Patterns of use of hemostatic agents in patients undergoing major surgery.
        J Surg Res. 2014; 186: 458-466
        • Tollec S.M.
        • Trossaert M.
        • Duveau D.
        • Grimandi G.
        • Sellal K.O.
        A proposed model for evaluating surgical hemostatic agents.
        Le Pharmacien Hospitalier et Clinicien. 2014; 49: 262-276
        • Murphy G.J.
        • Reeves B.C.
        • Rogers C.A.
        • Rizvi S.I.
        • Culliford L.
        • Angelini G.D.
        Increased mortality, postoperative morbidity and cost after red blood cell transfusion in patients having cardiac surgery.
        Circulation. 2007; 116: 2544-2552
        • Glance L.G.
        • Dick A.W.
        • Mukamel D.B.
        • Fleming F.J.
        • Zollo R.A.
        • Wissler R.
        • et al.
        Association between intraoperative blood transfusion and mortality and morbidity in patients undergoing noncardiac surgery.
        Anesthesiology. 2011; 114: 283-292
        • Dimick J.B.
        • Chen S.L.
        • Taheri P.A.
        • Henderson W.G.
        • Khuri S.F.
        • Campbell Jr., D.A.
        Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program.
        J Am Coll Surg. 2004; 199: 531-537
        • Shander A.
        Financial and clinical outcomes associated with surgical bleeding complications.
        Surgery. 2007; 142: S20-S25
        • Christensen M.C.
        • Krapf S.
        • Kempel A.
        • von Heymann C.
        Costs of excessive postoperative hemorrhage in cardiac surgery.
        J Thorac Cardiovasc Surg. 2009; 138: 687-693
        • Stokes M.E.
        • Ye X.
        • Shah M.
        • Mercaldi K.
        • Reynolds M.W.
        • Rupnow M.F.
        • et al.
        Impact of bleeding-related complications and/or blood product transfusions on hospital costs in inpatient surgical patients.
        BMC Health Serv Res. 2011; 11: 135
        • Levy J.H.
        • Dutton R.P.
        • Hemphill 3rd, J.C.
        • Shander A.
        • Cooper D.
        • Paidas M.J.
        • et al.
        Multidisciplinary approach to the challenge of hemostasis.
        Anesth Analg. 2010; 110: 354-364
        • Yamamoto K.
        • Wada H.
        • Sakakura K.
        • Ikeda N.
        • Yamada Y.
        • Katayama T.
        • et al.
        Cardiovascular and bleeding risk of non-cardiac surgery in patients on antiplatelet therapy.
        J Cardiol. 2014; 64: 334-338
        • Devereaux P.J.
        • Mrkobrada M.
        • Sessler D.I.
        • Leslie K.
        • Alonso-Coello P.
        • Kurz A.
        • et al.
        Aspirin in patients undergoing noncardiac surgery.
        N Engl J Med. 2014; 370: 1494-1503
        • Schuhmacher C.
        • Pratschke J.
        • Weiss S.
        • Schneeberger S.
        • Mihaljevic A.L.
        • Schirren R.
        • et al.
        Safety and effectiveness of a synthetic hemostatic patch for intraoperative soft tissue bleeding.
        Med Devices (Auckl). 2015; 8: 167-174
        • Fischer C.P.
        • Bochicchio G.
        • Shen J.
        • Patel B.
        • Batiller J.
        • Hart J.C.
        A prospective, randomized, controlled trial of the efficacy and safety of fibrin pad as an adjunct to control soft tissue bleeding during abdominal, retroperitoneal, pelvic, and thoracic surgery.
        J Am Coll Surg. 2013; 217: 385-393
        • United States Food and Drug Administration
        Clinical outcome assessment qualification program [Internet].
        US Food and Drug Administration, Silver Spring (MD)2016 (Available from:)
        • Swindle M.M.
        • Makin A.
        • Herron A.J.
        • Clubb Jr., F.J.
        • Frazier K.S.
        Swine as models in biomedical research and toxicology testing.
        Vet Pathol. 2012; 49: 344-356
        • Lesaffre A.R.
        Statistical and methodological aspects of oral health research.
        John Wiley & Sons Ltd, West Sussex (United Kingdom)2009
        • Schmidt R.C.
        Managing Delphi surveys using nonparametric statistical techniques.
        Decis Sci. 1997; 28: 763-774
        • Siegel J.M.
        • Cummings J.F.
        • Clymer J.W.
        Reproducible, repeatable and clinically-relevant hemostasis scoring.
        J Adv Med Pharm Sci. 2014; 1: 30-39
        • Ng C.S.
        • Pickens A.
        • Siegel J.M.
        • Clymer J.W.
        • Cummings J.F.
        A novel narrow profile articulating powered vascular stapler provides superior access and haemostasis equivalent to conventional devices.
        Eur J Cardiothorac Surg. 2016; 49: i73-i78
        • Jackman S.V.
        • Cadeddu J.A.
        • Chen R.N.
        • Micali S.
        • Bishoff J.T.
        • Lee B.R.
        • et al.
        Utility of the harmonic scalpel for laparoscopic partial nephrectomy.
        J Endourol. 1998; 12: 441-444
        • Adams G.L.
        • Manson R.J.
        • Hasselblad V.
        • Shaw L.K.
        • Lawson J.H.
        Acute in-vivo evaluation of bleeding with Gelfoam plus saline and Gelfoam plus human thrombin using a liver square lesion model in swine.
        J Thromb Thrombolysis. 2009; 28: 1-5
        • Lewis K.M.
        • Atlee H.D.
        • Mannone A.J.
        • Dwyer J.
        • Lin L.
        • Goppelt A.
        • et al.
        Comparison of two gelatin and thrombin combination hemostats in a porcine liver abrasion model.
        J Invest Surg. 2013; 26: 141-148
        • Lewis K.M.
        • Atlee H.
        • Mannone A.
        • Lin L.
        • Goppelt A.
        Efficacy of hemostatic matrix and microporous polysaccharide hemospheres.
        J Surg Res. 2015; 193: 825-830
        • Lewis K.M.
        • Spazierer D.
        • Urban M.D.
        • Lin L.
        • Redl H.
        • Goppelt A.
        Comparison of regenerated and non-regenerated oxidized cellulose hemostatic agents.
        Eur Surg. 2013; 45: 213-220
        • Lewis K.M.
        • McKee J.
        • Schiviz A.
        • Bauer A.
        • Wolfsegger M.
        • Goppelt A.
        Randomized, controlled comparison of advanced hemostatic pads in hepatic surgical models.
        ISRN Surg. 2014; 2014: 930803
        • Delilkan A.E.
        Comparison of subjective estimates by surgeons and anaesthetists of operative blood loss.
        Br Med J. 1972; 2: 619-621
        • Schorn M.N.
        Measurement of blood loss: review of the literature.
        J Midwifery Womens Health. 2010; 55: 20-27
        • Guinn N.R.
        • Broomer B.W.
        • White W.
        • Richardson W.
        • Hill S.E.
        Comparison of visually estimated blood loss with direct hemoglobin measurement in multilevel spine surgery.
        Transfusion. 2013; 53: 2790-2794
        • Adkins A.R.
        • Lee D.
        • Woody D.J.
        • White Jr., W.A.
        Accuracy of blood loss estimations among anesthesia providers.
        AANA J. 2014; 82: 300-306
        • Dildy 3rd, G.A.
        • Paine A.R.
        • George N.C.
        • Velasco C.
        Estimating blood loss: can teaching significantly improve visual estimation?.
        Obstet Gynecol. 2004; 104: 601-606
        • Lamhaut L.
        • Apriotesei R.
        • Combes X.
        • Lejay M.
        • Carli P.
        • Vivien B.
        Comparison of the accuracy of noninvasive hemoglobin monitoring by spectrophotometry (SpHb) and HemoCue® with automated laboratory hemoglobin measurement.
        Anesthesiology. 2011; 115: 548-554
        • Holmes A.A.
        • Konig G.
        • Ting V.
        • Philip B.
        • Puzio T.
        • Satish S.
        • et al.
        Clinical evaluation of a novel system for monitoring surgical hemoglobin loss.
        Anesth Analg. 2014; 119: 588-594
        • Konig G.
        • Holmes A.A.
        • Garcia R.
        • Mendoza J.M.
        • Javidroozi M.
        • Satish S.
        • et al.
        In vitro evaluation of a novel system for monitoring surgical hemoglobin loss.
        Anesth Analg. 2014; 119: 595-600
        • Sharareh B.
        • Woolwine S.
        • Satish S.
        • Abraham P.
        • Schwarzkopf R.
        Real time intraoperative monitoring of blood loss with a novel tablet application.
        Open Orthop J. 2015; 9: 422-426