Which AI systems do employers need to watch out for?

With the rise of various AI systems, HR departments are increasingly using them, for example, for recruitment, evaluating employee performance and similar purposes. However, the AI Act already prohibits the use of certain systems outright, while for others – so-called high-risk systems – it requires compliance with numerous conditions from December 2027. Below, we provide a brief summary of both these categories to help you better understand your obligations.

Prohibited systems

The definition and basic framework of so-called prohibited systems is set out in Article 5 AI Act. Of particular importance for HR is the prohibition of systems that:

• recognise emotions in the workplace,
• use a biometric categorisation system to deduce race, political opinions, trade union membership, religious or philosophical beliefs, or sex life or sexual orientation, or
• use a social scoring system (“social score”) of natural persons for the purpose of detrimental or unfavourable treatment.

The list is, however, considerably longer and further includes, for example, systems that use subliminal, manipulative or deceptive techniques, systems that exploit the vulnerabilities of natural persons, etc. Therefore, check which AI system you are using and verify whether it displays any features corresponding to the list in Article 5 AI Act.

High-risk systems

For high-risk systems, Article 6 AI Act uses a definition based on lists. Among HR systems, the following are high-risk:

• systems for the recruitment of employees, where they are used to select candidates, and to analyse and filter job applications and evaluate candidates,
• systems for the management of employees, where they are used to make decisions on promotion, termination of employment, the allocation of tasks based on an employee’s behaviour or characteristics, and to monitor and evaluate performance, or for any other decision affecting the terms of the employment relationships.

A high-risk system may generally be used, but it is necessary to comply with many relatively complex obligations laid down under Article 8 et seq. AI Act – e.g. a risk-management system, record-keeping and so on.

In the case of HR systems, however, it is possible to disapply most of these obligations. Systems referred to in Annex III (which include the HR systems mentioned above) are not, in fact, considered high-risk where interaction between a natural person and the AI system is ensured when the system is used – i.e. either the AI system enhances the work carried out by a particular person (e.g. an HR manager), or, conversely, that person reviews and finalises what the AI system prepares for them.

But beware! A system is always high-risk where it performs profiling of natural person. For such systems, disapplying the obligations is therefore not possible and all of them must be complied with.

The obligation to comply with all the conditions for the use of so-called high-risk systems in HR will apply from 2 December 2027, while the category of AI systems embedded in regulated products (to put it simply) will apply from 2 August 2028. We therefore recommend assessing which AI systems you use, which obligations you must comply with when using them, and, for HR systems, ensuring at least a sufficient element of human oversight.

And what penalties are you exposed to for breaching these obligations?

• For the use of prohibited practices, up to EUR 35 million, or up to 7 % of the total worldwide annual turnover of the undertaking, whichever is higher.
• For breaching the obligations relating to the use of high-risk systems, up to EUR 15 million or up to 3 % of the total worldwide annual turnover, whichever is higher.