代做program、代写Java/C++程序语言

日期：2024-12-07 10:30

Smart Industry Operations 2024-2025

Individual Assignment

Classification, Bias, and Fairness

Context of the Assignment

The context of this assignment is the use of predictive AI tools to support decision making in

operational settings. The specific operational setting is in the criminal justice system and the case of

interest is the COMPAS system, referred to in the lectures:

COMPAS: Correctional Offender Management Profiling for Alternative Sanction

The purpose of the assignment is to understand by applying in practice

- How classification methods work and how they can be appropriately assessed

- Understand aspects of bias in AI outcomes

- Make suggestions about how such bias can be mitigated.

The COMPAS system was operationalised in the United States and was provided to the states’

justice systems by private companies. The motivation for the justice system was to have a more

efficient system in place, which would also potentially avoid human bias in making a decision. The

decision is about whether to deny or grant an offender release from prison on the basis of a risk

assessment regarding whether the specific person was at risk of offence recidivism. The risk

assessment is based on a form such as:

www.documentcloud.org/documents/2702103-Sample-Risk-Assessment-COMPAS-CORE

The COMPAS system was designed to use collected data from such risk assessments to make a

prediction about whether an offender under consideration was predicted to be at an risk or not of

recidivism.

Case Data

To perform the tasks in this assignment, you are provided with real data from the operational use

of the COMPAS system. The data set is structured as follows:

age: A numerical variable, marking the age of the person

juv_fel_count: A numerical variable, marking the number of juvenile felonies

juv_misd_count: A numerical variable, counting the number of juvenile misdemeanours

juv_other_count: A numerical variable, indicating the number of juvenile convictions that were

neither felonies nor misdemeanors

priors_count: A numerical variable, providing the number of prior crimes committed

is_recid: Binary variable, stating whether the person recidivate within 2 years (1:yes, 0:no)?

sex: Categorical, the gender of the defendant (broken down in 2 dummy variables) race: Categorical, the race of the defendant, broken down in dummy variables

c_charge_degree: Categorical, the degree by which the person is currently being charged with. The

categories are: felonies, misdemeanors, and infractions, ordered from most serious to least. These

are further broken down to charge types (felonies or misdemeanours), and in each category from

the 1st (most serious offenses) to the least severe (F1-3, M1-MO3). Finally there is an attribute for

other offences and this is a binary attribute (0 or 1).

compas_score: This is the final prediction that the COMPAS system has made for each person,

being at risk or no of offence recidivism. 1: high/medium risk, 0: low risk.

The provided datasets are:

A. recidivism-risk.csv

A complete dataset, populated with all above information. You will use this for your analysis and for

classification modelling training purposes.

B. . recidivism-risk-predict.csv

This is a similar dataset to A, but on this one you have no access to the compass_score. You will use

this to predict the compass score (classify unknown cases).

Assignment Questions

A.1. Exploratory Data Analysis (10% of Assignment 1 mark)

In this part you are expected to:

A1.1. Explore the variables, their types, and their basic statistics.

A1.2. Analyse further the data regarding data distributions, range of values, existence of outliers

and correlations between attributes, as well as between input attributes and compass_score. To

what extent is the dataset balanced regarding the different categories of sex and race and across

the age ranges?

A.2. Classification (30% of Assignment 1 mark)

In this part you are expected to develop classifier models. You will have to consider how best to use

your training data (recidivism-risk.csv) and you are asked to apply the developed models to the

“recidivism-risk-predict.csv” data.

A2.1. Apply a decision tree classifier, choosing different tree depths on the recidivism-risk.csv data.

Motivate your solution analysis in relation to overfit and generalization. Report and analyse

performance using different performance metrics. Analyse your findings. Finally, choose a

developed model and apply it to the recidivism-risk-predict.csv data to produce your predictions.

A2.2. Apply a random forest classifier. random forest regression choosing different number of

estimators and tree depths. Motivate your solution and analysis in relation to overfit and

generalization. Report and analyse performance using different performance metrics. Analyse your

findings. Finally, choose a developed model and apply it to the recidivism-risk-predict.csv data to

produce your predictions. A2.3. Apply a Naïve Bayes classifier. Report and analyse performance using different performance

metrics. Analyse your findings. Finally, choose a developed model and apply it to the recidivismrisk-predict.csv

data to produce your predictions.

A2.4. Apply a support vector classifier. Report and analyse performance using different

performance metrics for three different ‘gamma’ values. Analyse your findings. Finally, choose a

developed model and apply it to the recidivism-risk-predict.csv data to produce your predictions.

A2.5. Make a comparative analysis across all classifier experiments. Make a reasoned choice of a

classifier to select and motivate the choice referring to the evidence obtained from performance

metrics.

A.3. Bias Analysis and Management (50% of Assignment 2 mark)

In this part you are expected to further analyse the data regarding potential bias. Specifically

consider the characteristics ‘age’, and sex (female, male), and race (all categories).

A3.1.1-A3.1.4. Perform exactly the same experiments as in part A.2. but without taking into account

the ‘race’ attributes. Analyse the difference of the obtained results compared to A.2. (A3.1.1 is for

decision tree, A3.1.2 for random forest, A3.1.3 for Naïve Bayes, A3.1.4 for support vector classifier).

Consider also your observations from part A.1. regarding the distribution of data in the recidivismrisk.csv

dataset with regard to the race attributes in your analysis.

A3.2.1-A3.2.4. Perform exactly the same experiments as in part A.2. but without taking into account

the ‘sex’ attributes. Analyse the difference of the obtained results compared to A.2. (A3.2.1 is for

decision tree, A3.2.2 for random forest, A3.2.3 for Naïve Bayes, A3.2.4 for support vector classifier).

Consider also your observations from part A.1. regarding the distribution of data in the recidivismrisk.csv

dataset with regard to the sex attributes in your analysis.

A3.3.1-A3.3.4. Perform exactly the same experiments as in part A.2. but without taking into account

the ‘sex’ and ‘race’ attributes. Analyse the difference of the obtained results compared to A.2.

(A3.3.1 is for decision tree, A3.3.2 for random forest, A3.3.3 for Naïve Bayes, A3.3.4 for support

vector classifier). Consider also your observations from part A.1. regarding the distribution of data

in the recidivism-risk.csv dataset with regard to the sex attributes in your analysis.

A.4. Overall comparisons and analysis (10% of Assignment 2 mark))

In this part you are expected to:

Discuss comparatively the obtained results highlight only what you see as most interesting

regarding the obtained performance and/or aspects of data unbalance, and fairness, motivating

your analysis on the basis of the obtained evidence. What would be your concluding

recommendations?.

A.5. Bias Management and Mitigation (up to 15% extra of Assignment 2 mark, capped to

maximum assignment mark)

This is a bonus part of the assignment for the teams that aim to ensure a very high mark. In this

part you are free to work creatively on the basis of what you have seen in the lectures and

practicals. The aim in this part is to apply bias management and mitigation methods as applicable to

either the race or the sex attributes in the dataset. You are not expected to do this exhaustively but

to work on limited experiments of your choice (for example regarding one of the two sensitive attributes, sex or race). The aim of this extra ‘question’ is to trigger you into creative thinking and

action, rather than instruct you to perform a very specific task.

A general remark on grading: Note that the quality of your analysis in each step will be taken into

account. Make sure your code is well-documented and your report is readable, such that what is

done is clear and motivated. We will evaluate your contribution based on what is explained and

documented in your Jupyter notebook report.