When algorithms decide your rights

Then she remembered her appointment and rushed out of bed: 9:30am at her local Citizens Advice Bureau. To get from Chalkhill Estate to the High Road she would need to run and have some luck to catch the bus on time.

In the bus on the way to her appointment, she felt like this journey had been in the making for a long time. Four years ago, her sister Chantell, who had cerebral palsy and heavily relied on support, had her home care visits dramatically cut from 56 to 32 hours a week. A new algorithm had reassessed the amount of care her sister would be given. Her sister had pleaded with the assessor, explaining how that simply wasn’t enough support, but neither the assessor nor her sister seemed to quite understand how the decision was reached by the computer to reduce the amount of care. Her sister’s health situation hadn’t improved, but an invisible change had occurred that created this new result. When the assessor entered the information about her health status, daily routines and needs for support into the computer, it ran through an algorithm that Brent council had recently approved, determining how many hours of help she would receive.

And then there was her younger brother Jordan who had been arrested and charged with burglary and petty theft for grabbing an unlocked bike and a scooter with his mate. When Jordan was booked into prison, a computer program spat out a score predicting the likelihood of him committing a future crime. Yes, Jordan had had issues before and a criminal record for misdemeanours committed when he was a juvenile. But how could he be classified at a high risk of re-offending? He had told her that so many other seasoned criminals with multiple convictions of armed robbery had been classified as low risk. But then those guys were white and Jordan was black…

So now it was her turn. Yes she was not the perfect mum, she was the first to admit that herself. She was struggling, not just because of her learning disability which made it hard to stay in a job, but also because she tried to help her sister after home visits were cut.

Unfortunately, her energy to support Selena and Brandon was often nil and they regularly missed school. So a few weeks ago a woman from the council had come by her house and had told her that her family was classified as high-risk and was being placed in a special programme of families being at risk of child sexual abuse and gang exploitation. She had been horrified to hear this and needed help. Her neighbour Sue had told her that Citizens Advice had launched a new service: AAS – the algorithm advice service.

Fred, the young Citizens Advisor, was a student training as a data scientist. He would help her to analyse which data points had triggered her high-risk classification and what rights she had to contest some of the data used by the council and the conclusions drawn.