Quality control

Both the imaging and the clinical/behavioral data have undergone quality controls to ensure a high quality product.

Several levels of quality controls for the imaging data have been implemented. The first level was conducted at the site immediately following data acquisition. Once data arrived at the Data Coordinating Center (DCC), it was archived and then converted to MINC, the medical image file format developed at the Brain Imaging Centre at Montreal Neurological Institute. Here, some automated quality control took place to ensure correct labeling and proper acquisition parameters. MRI data then underwent a further series of quality control measures. For more details about MRI quality control, please click here.

Several levels of quality controls for the collection of clinical/behavioral data were implemented. Initially, examiners and interviewers were trained on standardized administration and scoring procedures for the various instruments. Periodically throughout data collection, regular reviews of audiotapes, videotapes, and completed test forms were conducted by the Clinical Coordinating Center (CCC) to ensure proper administration and scoring. Once data was transferred to the DCC, automated and visual quality controls were implemented. Please click here for a summary of CCC and DCC quality control procedures. For additional details, see the study protocol, procedure manuals, and published methods papers (Evans, et al., 2006; Almli, et al., 2007).

Disclaimer

While every effort was made to ensure the accuracy of the data, it is possible that errors exist. NIH and the study investigators do not and cannot warrant the results that may be obtained by using the data. NIH and the study investigators disclaim all warranties as to the accuracy of the data in the study database or the performance or fitness of the data for any particular purpose.

Description of MRI quality control procedures

The first level of quality control was conducted at the site, immediately following the data acquisition. Image files were transferred from the console to a Study Work Station (SWS) where they were reviewed using visualization software. Data was then transferred to the Data Coordinating Center (DCC) in Montreal.

Data arrived at the DCC in DICOM format. This data was archived and then converted to MINC, the medical image file format developed at the Brain Imaging Centre. The two assigned identification numbers (labels) were checked to make sure that they matched across all volumes sent for the particular subject. The acquisition parameters for each volume for the subject were checked against the scanning protocol automatically. If there was a problem with the identification labels, the site was contacted and asked to relabel and resend the data. Once the labels passed verification, the data was inserted into the database with a status flag set to PENDING for the subject and for each image volume. The MRI data were then ready for a review of the image quality.

A verification image (sample slice of the full image) was created from the native data containing a mid-transverse, sagittal and coronal image through the volume. These were the images used to organize the visual inspection described below.

For each volume, an automatic program ran a number of scripts on all of the selected slices to accomplish the following tasks:

• A histogram of the volume was computed.
• Movement artifacts were estimated.
• Noise and contrast were estimated.
• Image preprocessing (cropping, image intensity non-uniformity correction, intensity normalization) was completed.

Using the acquisition parameters, the volumes for T1, T2 and PD-weighted anatomical were pre-selected. Once completed, the data was ready for visual inspection.

The goal of the visual inspection was to rate:

• the amount of movement artifacts – either within the slices or volume, or between packets for the multi-packet acquisitions.
• the level of intensity homogeneity within slices, between slices and throughout the volume (the amount of noise in the scan).
• the level of contrast between gray matter, white matter and CSF.
• the adherence to the scanning protocol in terms of coverage of the head/brain from left to right, top to bottom and front to back.
• the amount of geometric distortion due to susceptibility artifacts.
• the appearance of any other artifacts in the images.

Individual volumes were given a ”PASS” or “FAIL” quality control status, based on the categories above.

Passing MRI datasets: The complete dataset for a subject visit was then given a “PASS” or “FAIL” status determined by the successful acquisition of the structural MRIs (T1W, PD/T2W; either full protocol or fallback protocol [used where a limited protocol was necessitated by the level of subject compliance]). A dataset must have had a successful T1W and a PD/T2W to have received a final “PASS” status. If a scan was failed, a rescan was requested (assuming there was time remaining within the allowable age window).

Description of clinical/behavioral quality control procedures at the Clinical Coordinating Center (CCC)

The following summarizes of the clinical/behavioral quality control procedures that were implemented at the CCC:

1. All quality control materials (e.g., videotapes/audiotapes, paper copies of completed test booklets/score sheets, questionnaire forms, etc.) from the Pediatric Study Centers were sent to the CCC in a timely fashion, i.e., as soon as possible after completion of testing and scoring, but no later than two weeks after testing.

2. The sampled materials were logged, tracked, and monitored by the CCC.

3. In some instances, portions of the reviews were accomplished at a collaborating site based on its expertise with specific instruments. In such cases, the CCC provided the site with the materials to be reviewed.

4. Each instrument was rated as:

a. Passing -defined as ≥ 90% agreement with the standard for item administration and scoring (required for valid data when testing real ‘scanned’ subjects);
b. Provisionally passing -Minor problems, potentially passing (may or may not yield valid data); or
c. Administered/scored incorrectly--Major problems, < 90% agreement with the standard for item administration and scoring (invalid data when testing real ‘scanned’ subjects).

5. Written feedback about administration and scoring performance was provided to each rater/tester by the evaluator. This feedback was provided in the form of a checklist review with space for specific comments. The written feedback was also forwarded to site’s behavioral investigator and principal investigator (as well as any others designated by the principal investigator), and the CCC.

6. Copies of the evaluation checklists and comments were retained at the CCC.

7. The CCC entered all quality control related data into a database to consolidate results for monitoring the overall process.

8. The CCC also entered individual evaluation results into the Examiner Certification fields of the DCC database. These fields were used to ‘flag’ the performance of individual testers/raters and their associated testing data.

9. Technically incomplete, insufficient or poor video/audio recordings (e.g., image or sound not adequate for accurate evaluation, missing tests or parts of tests) of testing were not reviewed. Such situations were rated as “No quality control decision,” and the tester/rater had to redo the testing and recording with additional practice children (i.e., non-subjects) for submission to the CCC for evaluation.

10. If “correctable” errors (e.g., certain scoring errors) were noted during the evaluation process, the rater/tester was required to correct the error(s) on the score sheets/booklets and in the DCC database. The tester/rater notified the evaluator by sending the corrected score sheets/booklets back to the evaluator (via FAX or some form of express mail), who confirmed that the correction(s) had been made in the DCC database. If a subject’s data profile had already been sent to the DCC, the tester/rater contacted the DCC to request access to the subject’s data so that corrections could be made. Failure to send corrected materials to the evaluator and to make required corrections in the DCC database resulted in the test (or subtest, as appropriate) being rated, “Administered/scored incorrectly.” The time allotted to complete this process was one week following notification, barring extenuating circumstances that were approved by the CCC.

Description of clinical/behavioral quality control at the Data Coordinating Center (DCC)

Clinical/behavioral quality control of the entry of each subject’s data into the database was centralized at the Data Coordinating Center (DCC), following the steps outlined below:

1. Staff at each Pediatric Study Center entered data using a standard laptop provided to each site and uploaded the data into the DCC database.

2. DCC requested a hard copy of paper and pencil forms for which data was hand entered using a random schedule.

• The list of hard copies requested was made available to each site via an RSS Channel link.
  
• For objective 1, visits 1 and 2, and objective 2 subjects, one in three subjects' forms was randomly selected. For objective 1, visit 3, one in six subjects' profiles was randomly selected for review.

3. Once a hard copy of the subject’s data was received at the DCC, the subject's data was moved into a review and approval stage, during which sites had ‘Read Only’ privileges. The entered data was confirmed against the hard copies.

4. When necessary, data corrections were requested using a web-based system. The sites were given ‘Read/Write’ access to those instruments for which data corrections were needed. Feedback identified clerical, input, and scoring inconsistencies or errors. Potential scoring errors encountered over the course of the DCC clinical/behavioral quality control were brought to the attention of the Clinical Coordinating Center (CCC).

5. Upon completion of corrections, the subject's data became ‘Read Only’ and the subject’s profile was tagged as having been ‘Hard copy quality controlled.’ The DCC maintained a log of the errors identified and the feedback interactions with each Pediatric Study Center.