Garbage in, garbage out; city watchdog finds troubling pattern of unreliable data quality

Just as Major League Baseball uses statistics to measure player performance and devise game-day strategy, cities use data to monitor employees and deliver services equitably and economically.

But what happens if the quality of that information is unreliable?

That’s the question being raised by the latest advisory from Chicago’s Acting Inspector General William Marback.

Marback identified a pattern of “data quality issues” impacting the “objectivity, utility and integrity across various city departments” of the information Chicago uses to allocate resources, measure employee performance and monitor a host of programs.

Among the problems:

• A 2021 audit of the Chicago Fire Department found only 75.2% of records for events over a nearly two-year period included all data necessary to reliably measure emergency response times or calculate “turnout and travel times for first-arriving units.” Remaining records contained “date-time milestones that were blank or out of sequence.” CFD brass “had been aware of data reliability issues since at least 2013, but had not remedied them.”

• Data quality and management issues with the personnel and payroll systems used by the Department of Human Resources. As of August 2020, 13% of employee leave records “lacked information about the basis for the leave or showed a reason that was not authorized” by city personnel rules. Records also “listed employees in departments that no longer existed” or “lacked essential information, including gender, race, bargaining unit,” valid addresses and zip codes.

• A 2020 advisory concluded the inventory of roughly 7,000 municipal license plates maintained by the Department of Assets, Information and Services did not match the database maintained by the Illinois Secretary of State’s office. Unaccounted for plates and vehicles “create the risk of abuse and misconduct with the city’s fleet,” Marback wrote.

• Problems with the mobile e-ticketing system the Department of Streets and Sanitation uses made it difficult or impossible for ward superintendents to enforce the requirement that businesses and residences with five or more units hire private haulers to haul away recycled refuse.

• Another 2020 advisory concluded Streets and Sanitation could not measure performance or meet its own “timely weed-cutting goals” because it “did not have an accurate list of city-owned lots” and still used paper records that were “sometimes lost” to pinpoint lots requiring maintenance. Streets and San employees also “did not consistently and accurately enter” those paper records into the service request tracking system.

• A 2020 audit concluded the private contractor hired to manage Chicago’s Juvenile Intervention and Support Center did not keep “accurate and consistent” records and used three different tracking systems to record program data — and that each system was “incomplete” or rife with “inaccuracies.” The shoddy records “made it impossible to determine whether the program was achieving positive or negative outcomes for the over 3,000 youths it served each year.”

• A 2020 review concluded processes used by the Chicago Police Department to manage and produce records to meet “constitutional and legal obligations” to respond to subpoenas or records requests were grossly inadequate. “CPD could not determine what records existed for any case or incident.” Nor did the department “track its production of records” or “determine where it stored paper records.”

• A widely-publicized 2019 review of Chicago’s error-filled gang database concluded CPD “lacked control over the generation, maintenance and sharing” of the sensitive information, did not “require users to provide evidence supporting a gang designation” and that “birth dates, gang affiliations and reasons for gang designation” within the data “contradicted each other or were missing entirely.” The gang database is in the process of being replaced.

• A 2019 audit of housing quality inspections conducted for Chicago’s Low-Income Housing Trust Fund showed those records “did not maintain complete documentation of known lead hazards, building code violation or local court action against subsidized properties.” As a result, it was impossible to “ensure subsidized properties were safe.”

• A 2019 audit of the Chicago Department of Transportation’s driveway billing process found “incomplete and inaccurate permit data” cost the city up to $1.5 million a year, probably more, because CDOT had no idea whether it had “recorded all relevant driveways.”

“Governments that actively use data can evaluate the success of public service delivery and engage in continuous improvement. But data can only serve its purposes if it is complete, accurate and reliable,” Marback wrote.

In response to his request for a “proactive culture of data quality management, Marback said, “The city’s chief data officer has outlined current and future data initiatives and we are encouraged by the ongoing effort to ensure that the data issues that frequently come up … will be addressed going forward.”