Loss Functions
Introduction¶
Previously, we've gone through the basics in Part 1 of this demo series (CF ensemble using ALS). In this demo, we'll shift our focus toward experimental methods including the optimization using Tensorflow, designing neural architecture for CF ensemble learning, and cutomized loss functions.
CFNet¶
- Key operations
- Computing user and item embeddings
- Taking dot product followed by a non-linear transformation (e.g. sigmoid)
- We'll also introduce
cf_modelsmodule: A collection of NN-based models for CF ensemble and related utility functions
Import Libraries¶
First, let's set up the enviornment and import related libraries ...
In [ ]:
Copied!
# install openMP (as a prerequisite prior to installing faiss)
!sudo apt-get install libomp-dev
# => doing so allows for "pip install faiss"
# => which then also allows for "import utils_knn"
# NOTE:
# kNN-based methods won't be discussed until "Part 3" but we will have a bit of
# a spoiler alert here in order to use kNN to estimate labels in `T`
# install openMP (as a prerequisite prior to installing faiss)
!sudo apt-get install libomp-dev
# => doing so allows for "pip install faiss"
# => which then also allows for "import utils_knn"
# NOTE:
# kNN-based methods won't be discussed until "Part 3" but we will have a bit of
# a spoiler alert here in order to use kNN to estimate labels in `T`
Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: libnvidia-common-460 nsight-compute-2020.2.0 Use 'sudo apt autoremove' to remove them. The following additional packages will be installed: libomp5 Suggested packages: libomp-doc The following NEW packages will be installed: libomp-dev libomp5 0 upgraded, 2 newly installed, 0 to remove and 42 not upgraded. Need to get 239 kB of archives. After this operation, 804 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libomp5 amd64 5.0.1-1 [234 kB] Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libomp-dev amd64 5.0.1-1 [5,088 B] Fetched 239 kB in 1s (454 kB/s) debconf: unable to initialize frontend: Dialog debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 2.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (This frontend requires a controlling tty.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Selecting previously unselected package libomp5:amd64. (Reading database ... 155203 files and directories currently installed.) Preparing to unpack .../libomp5_5.0.1-1_amd64.deb ... Unpacking libomp5:amd64 (5.0.1-1) ... Selecting previously unselected package libomp-dev. Preparing to unpack .../libomp-dev_5.0.1-1_amd64.deb ... Unpacking libomp-dev (5.0.1-1) ... Setting up libomp5:amd64 (5.0.1-1) ... Setting up libomp-dev (5.0.1-1) ... Processing triggers for libc-bin (2.27-3ubuntu1.3) ... /sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link
In [ ]:
Copied!
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
import os, sys
# Colab
try:
import google.colab
IN_COLAB = True
except:
IN_COLAB = False
# Plotting
import matplotlib.pylab as plt
# %matplotlib inline
from matplotlib.pyplot import figure
import seaborn as sns
from IPython.display import display
# Progress
from tqdm import tqdm
################################################################
# Configure system environment
# - Please modify input_dir according to your local enviornment
#
################################################################
cur_dir = os.getcwd()
project_dir = 'machine_learning_examples/cf_ensemble'
if IN_COLAB:
# Run this demo on Google Colab
from google.colab import drive
drive.mount('/content/drive')
# Parameters for data
input_dir = f"/content/drive/MyDrive/Colab Notebooks/{project_dir}"
# /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/data/data-is-life
sys.path.append(input_dir)
else:
input_dir = cur_dir
if input_dir != cur_dir:
sys.path.append(input_dir)
print(f"> Adding {input_dir} to sys path ...")
print(sys.path)
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
import os, sys
# Colab
try:
import google.colab
IN_COLAB = True
except:
IN_COLAB = False
# Plotting
import matplotlib.pylab as plt
# %matplotlib inline
from matplotlib.pyplot import figure
import seaborn as sns
from IPython.display import display
# Progress
from tqdm import tqdm
################################################################
# Configure system environment
# - Please modify input_dir according to your local enviornment
#
################################################################
cur_dir = os.getcwd()
project_dir = 'machine_learning_examples/cf_ensemble'
if IN_COLAB:
# Run this demo on Google Colab
from google.colab import drive
drive.mount('/content/drive')
# Parameters for data
input_dir = f"/content/drive/MyDrive/Colab Notebooks/{project_dir}"
# /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/data/data-is-life
sys.path.append(input_dir)
else:
input_dir = cur_dir
if input_dir != cur_dir:
sys.path.append(input_dir)
print(f"> Adding {input_dir} to sys path ...")
print(sys.path)
Mounted at /content/drive > Adding /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble to sys path ... ['', '/content', '/env/python', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.7/dist-packages/IPython/extensions', '/root/.ipython', '/content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble', '/content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble']
In [ ]:
Copied!
# Tensorflow
import tensorflow as tf
print(tf.__version__)
# import tensorflow_probability as tfp
# tfd = tfp.distributions
from tensorflow import keras
# from tensorflow.keras import layers
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Flatten, Dense, Dropout, Lambda, Embedding
from tensorflow.keras.optimizers import RMSprop
from keras.utils.vis_utils import plot_model
from tensorflow.keras import backend as K
#################################################################
# Scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold, cross_val_predict, cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, StackingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
#################################################################
# CF-ensemble-specific libraries
import utils_stacking as ustk
import utils_classifier as uclf
import utils_sys as us
import utils_cf as uc
import scipy.sparse as sparse
from utils_sys import highlight
#################################################################
# try:
# import faiss
# except:
# # pip install faiss
# usys.install('faiss')
# import faiss
# Misc
import pprint
import tempfile
from typing import Dict, Text
np.set_printoptions(precision=3, edgeitems=5, suppress=True)
# Tensorflow
import tensorflow as tf
print(tf.__version__)
# import tensorflow_probability as tfp
# tfd = tfp.distributions
from tensorflow import keras
# from tensorflow.keras import layers
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Flatten, Dense, Dropout, Lambda, Embedding
from tensorflow.keras.optimizers import RMSprop
from keras.utils.vis_utils import plot_model
from tensorflow.keras import backend as K
#################################################################
# Scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold, cross_val_predict, cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, StackingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
#################################################################
# CF-ensemble-specific libraries
import utils_stacking as ustk
import utils_classifier as uclf
import utils_sys as us
import utils_cf as uc
import scipy.sparse as sparse
from utils_sys import highlight
#################################################################
# try:
# import faiss
# except:
# # pip install faiss
# usys.install('faiss')
# import faiss
# Misc
import pprint
import tempfile
from typing import Dict, Text
np.set_printoptions(precision=3, edgeitems=5, suppress=True)
2.8.0
Generate some data to play with¶
In [ ]:
Copied!
%matplotlib inline
from sklearn import datasets
from sklearn.datasets import make_classification
from collections import Counter
# get the dataset
c_ratio = 0.99
def get_dataset(n_samples=5000, noise=True):
if noise:
X,y = make_classification(n_samples=n_samples, n_features=100, n_informative=30,
n_redundant=6, n_repeated=3, n_classes=2, n_clusters_per_class=1,
class_sep=2,
flip_y=0.2, # <<<
weights=[c_ratio, ], random_state=17)
else:
X,y = make_classification(n_samples=n_samples, n_features=100, n_informative=30,
n_redundant=6, n_repeated=3, n_classes=2, n_clusters_per_class=1,
class_sep=2,
flip_y=0, weights=[c_ratio, ], random_state=17)
return X, y
X, y = get_dataset(noise=True)
# Plot data
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
sns.scatterplot(X[:,0],X[:,1],hue=y,ax=ax1);
ax1.set_title("With Noise");
plt.show();
uniq_labels = np.unique(y)
n_classes = len(uniq_labels)
# Turn into a binary classification problem
if n_classes > 2:
print('> y before:\n', y)
y, y_map, le = uclf.to_binary_classification(y, target_class=2)
print('> y after:\n', y)
print(f'> n_classes: {n_classes}\n{uniq_labels}\n')
counter = Counter(y)
print(f'> counts:\n{counter}\n')
%matplotlib inline
from sklearn import datasets
from sklearn.datasets import make_classification
from collections import Counter
# get the dataset
c_ratio = 0.99
def get_dataset(n_samples=5000, noise=True):
if noise:
X,y = make_classification(n_samples=n_samples, n_features=100, n_informative=30,
n_redundant=6, n_repeated=3, n_classes=2, n_clusters_per_class=1,
class_sep=2,
flip_y=0.2, # <<<
weights=[c_ratio, ], random_state=17)
else:
X,y = make_classification(n_samples=n_samples, n_features=100, n_informative=30,
n_redundant=6, n_repeated=3, n_classes=2, n_clusters_per_class=1,
class_sep=2,
flip_y=0, weights=[c_ratio, ], random_state=17)
return X, y
X, y = get_dataset(noise=True)
# Plot data
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
sns.scatterplot(X[:,0],X[:,1],hue=y,ax=ax1);
ax1.set_title("With Noise");
plt.show();
uniq_labels = np.unique(y)
n_classes = len(uniq_labels)
# Turn into a binary classification problem
if n_classes > 2:
print('> y before:\n', y)
y, y_map, le = uclf.to_binary_classification(y, target_class=2)
print('> y after:\n', y)
print(f'> n_classes: {n_classes}\n{uniq_labels}\n')
counter = Counter(y)
print(f'> counts:\n{counter}\n')
> n_classes: 2
[0 1]
> counts:
Counter({0: 4465, 1: 535})
Setting up the CF ensemble learner¶
In [ ]:
Copied!
# Create Base Learners
base_learners = [
('RF', RandomForestClassifier(n_estimators= 200,
oob_score = True,
class_weight = "balanced",
random_state = 20,
ccp_alpha = 0.1)),
('KNNC', KNeighborsClassifier(n_neighbors = len(np.unique(y))
, weights = 'distance')),
# ('SVC', SVC(kernel = 'linear', probability=True,
# class_weight = 'balanced'
# , break_ties = True)),
('GNB', GaussianNB()),
('QDA', QuadraticDiscriminantAnalysis()),
('MLPClassifier', MLPClassifier(alpha=1, max_iter=1000)),
# ('DT', DecisionTreeClassifier(max_depth=5)),
# ('GPC', GaussianProcessClassifier(1.0 * RBF(1.0))),
]
# Create Base Learners
base_learners = [
('RF', RandomForestClassifier(n_estimators= 200,
oob_score = True,
class_weight = "balanced",
random_state = 20,
ccp_alpha = 0.1)),
('KNNC', KNeighborsClassifier(n_neighbors = len(np.unique(y))
, weights = 'distance')),
# ('SVC', SVC(kernel = 'linear', probability=True,
# class_weight = 'balanced'
# , break_ties = True)),
('GNB', GaussianNB()),
('QDA', QuadraticDiscriminantAnalysis()),
('MLPClassifier', MLPClassifier(alpha=1, max_iter=1000)),
# ('DT', DecisionTreeClassifier(max_depth=5)),
# ('GPC', GaussianProcessClassifier(1.0 * RBF(1.0))),
]
Run CF Stacker¶
In [ ]:
Copied!
from sklearn.metrics import f1_score
n_iter = 1
cf_stackers = []
for i in range(n_iter):
# Initialize CF Stacker
print(f"[info] Instantiate CFStacker #[{i+1}] ...")
clf = ustk.CFStacker(estimators=base_learners,
final_estimator=LogisticRegression(),
work_dir = input_dir,
fold_number = i, # use this to index traing and test data
verbose=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
clf.fit(X_train, y_train)
X_meta_test = clf.transform(X_test)
print(f"[info] shape(X_meta_test): {X_meta_test.shape}")
y_pred = clf.predict(X_test)
perf_score = f1_score(y_test, y_pred) # clf.score(X_test, y_test)
print('[result]', perf_score)
# Add test label for the convenience of future evaluation after applying a CF ensemble method
clf.cf_write(dtype='test', y=y_test)
# keep track of all the stackers (trained on differet parts of the same data as in CV or resampling)
cf_stackers.append(clf)
from sklearn.metrics import f1_score
n_iter = 1
cf_stackers = []
for i in range(n_iter):
# Initialize CF Stacker
print(f"[info] Instantiate CFStacker #[{i+1}] ...")
clf = ustk.CFStacker(estimators=base_learners,
final_estimator=LogisticRegression(),
work_dir = input_dir,
fold_number = i, # use this to index traing and test data
verbose=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
clf.fit(X_train, y_train)
X_meta_test = clf.transform(X_test)
print(f"[info] shape(X_meta_test): {X_meta_test.shape}")
y_pred = clf.predict(X_test)
perf_score = f1_score(y_test, y_pred) # clf.score(X_test, y_test)
print('[result]', perf_score)
# Add test label for the convenience of future evaluation after applying a CF ensemble method
clf.cf_write(dtype='test', y=y_test)
# keep track of all the stackers (trained on differet parts of the same data as in CV or resampling)
cf_stackers.append(clf)
[info] Instantiate CFStacker #[1] ...
(BaseCF) base est | name: RF, estimator: RandomForestClassifier(ccp_alpha=0.1, class_weight='balanced', n_estimators=200,
oob_score=True, random_state=20)
(BaseCF) base est | name: KNNC, estimator: KNeighborsClassifier(n_neighbors=2, weights='distance')
(BaseCF) base est | name: GNB, estimator: GaussianNB()
(BaseCF) base est | name: QDA, estimator: QuadraticDiscriminantAnalysis()
(BaseCF) base est | name: MLPClassifier, estimator: MLPClassifier(alpha=1, max_iter=1000)
(BaseCF) Base predictors:
[1] RF: RandomForestClassifier(ccp_alpha=0.1, class_weight='balanced', n_estimators=200,
oob_score=True, random_state=20)
[2] QDA: QuadraticDiscriminantAnalysis()
[3] MLPClassifier: MLPClassifier(alpha=1, max_iter=1000)
[4] KNNC: KNeighborsClassifier(n_neighbors=2, weights='distance')
[5] GNB: GaussianNB()
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 28.3s finished [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.3s finished [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.0s finished [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.2s finished [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 29.7s finished
[info] Saving X_meta (shape=(3750, 5)) at: /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble/data/train-0.npz [info] Saving X_meta (shape=(1250, 5)) at: /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble/data/test-0.npz [info] shape(X_meta_test): (1250, 5) [info] Saving X_meta (shape=(1250, 5)) at: /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble/data/test-0.npz [result] 0.0851063829787234 (cf_write) Adding new attribute y: [0 0 0 0 0 ... 0 1 0 0 0] ... (cf_write) Saving X_meta at: /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble/data/test-0.npz
To speed things up, we could assume that the previous cell was executed at least once so that the necessary level-1 dataset has been generated (probabilty matrix from base classifiers, etc.).
- Uncomment below if you wish to bypass the base predictor training step above
In [ ]:
Copied!
# fold_number = 0
# # Instantiate a new CFStacker if you do not wish to spend time training base classifiers above
# # ... this is ok assuming that pre-trained level-1 datasets are available under ./data
# print("[info] Instantiating a new instance of CFStacker bypassing training loop (assuming pre-trained data available) ...")
# clf = ustk.CFStacker(estimators=base_learners,
# final_estimator=LogisticRegression(), # <<< this is to be changed into a CF ensemble method
# work_dir = input_dir,
# fold_number = fold_number, # Use this to index traing and test data
# verbose=1)
# Or simply use the static method without even creating an instance of CFStacker
# ustk.CFStacker.cf_fetch2(fold_number=fold_number)
# fold_number = 0
# # Instantiate a new CFStacker if you do not wish to spend time training base classifiers above
# # ... this is ok assuming that pre-trained level-1 datasets are available under ./data
# print("[info] Instantiating a new instance of CFStacker bypassing training loop (assuming pre-trained data available) ...")
# clf = ustk.CFStacker(estimators=base_learners,
# final_estimator=LogisticRegression(), # <<< this is to be changed into a CF ensemble method
# work_dir = input_dir,
# fold_number = fold_number, # Use this to index traing and test data
# verbose=1)
# Or simply use the static method without even creating an instance of CFStacker
# ustk.CFStacker.cf_fetch2(fold_number=fold_number)
In [ ]:
Copied!
import data_pipeline as dp
import polarity_models as pmodel
# Load pre-trained level-1 data (associated with a given fold number)
fold_number = 0
R, T, U, L_train, L_test = dp.load_pretrained_level1_data(clf=clf, fold_number=fold_number, verbose=1)
n_train = R.shape[1]
# Estimate probability thresholds according to `fmax`
policy_threshold = 'balanced'
p_threshold = uc.estimateProbThresholds(R, L=L_train, pos_label=1, policy=policy_threshold)
# Remember to use "estimated labels" for the test set; not the true label `L_test` that we are trying to predict
# Method #1: Majority vote
lh = uc.estimateLabels(T, p_th=p_threshold) # We cannot use L_test (cheating), but we have to guesstimate
# Method #2: kNN
# Pc, Lh = pmodel.color_matrix(R, L_train, p_threshold)
# lh = uc.estimateLabelsByRanking(R, T, L_train, Pc, topn=3) # NOTE: tricky to make faiss library work here
L = np.hstack((L_train, lh))
X = np.hstack((R, T))
assert len(U) == X.shape[0]
print(f"> shape(R):{R.shape} || shape(T): {T.shape} => shape(X): {X.shape}")
import data_pipeline as dp
import polarity_models as pmodel
# Load pre-trained level-1 data (associated with a given fold number)
fold_number = 0
R, T, U, L_train, L_test = dp.load_pretrained_level1_data(clf=clf, fold_number=fold_number, verbose=1)
n_train = R.shape[1]
# Estimate probability thresholds according to `fmax`
policy_threshold = 'balanced'
p_threshold = uc.estimateProbThresholds(R, L=L_train, pos_label=1, policy=policy_threshold)
# Remember to use "estimated labels" for the test set; not the true label `L_test` that we are trying to predict
# Method #1: Majority vote
lh = uc.estimateLabels(T, p_th=p_threshold) # We cannot use L_test (cheating), but we have to guesstimate
# Method #2: kNN
# Pc, Lh = pmodel.color_matrix(R, L_train, p_threshold)
# lh = uc.estimateLabelsByRanking(R, T, L_train, Pc, topn=3) # NOTE: tricky to make faiss library work here
L = np.hstack((L_train, lh))
X = np.hstack((R, T))
assert len(U) == X.shape[0]
print(f"> shape(R):{R.shape} || shape(T): {T.shape} => shape(X): {X.shape}")
[info] list of base classifiers: ['RF' 'KNNC' 'GNB' 'QDA' 'MLPClassifier'] (estimateProbThresholds) policy: fmax [info] probability thresholds: [0.499 0. 0.008 0. 0.007] > shape(R):(5, 3750) || shape(T): (5, 1250) => shape(X): (5, 5000)
At this point, we have all the key matrices ready. They are:¶
R: Probability/rating matrix for the training split, in whichR[i][j]is the conditional probability output (i.e. P(y=1|X)) for the i-th base classifier ("user") predicting j-th training instance ("item").T: Probability/rating matrix for the test split.X: A column-stacked matrix combiningRandTinto a single matrix.U: Names of users/classifiers in the order consistent withaxis=0or row order of the rating matrices likeXp_threshold: Probability thresholds associated with the base classifiers. Each base classifier has its own threshold that optimizes a given performance matrix (E.g., the best probability threshold that reaches the best F1 score is called Fmax).L_train: True labels for the training splielh: Estimated labels for the test split (we need this to establish the optimization that derives latent factors)L_test: True labels for the test split (what we aim to predict)
Confidence Matrix¶
A mentioned in Part 1, a confidence matrix (C) is a matrix comprising confidence measures: C[i][j] is a confidence measure for i-th base classifier's prediction on j-th data point. Confidence matrix can come in a few different forms:
C0: The (raw) confidence matrix whose rows and columns are consistent with those of the probabilty/rating matrix (R,TorX).C0holds the confidence scores prior to any masking and re-weighting operations.Cw: Re-weighted confidence matrix (without masking). Seeutils_cf.balance_and_rescale()for more details.Cwis a "dense" confidence matrix, meaning that no "masking" is applied; that is, confidence scores for TPs, TNs and even FPs, FNs are all preserved.Cn: Masked confidence matrix in which the confidence measures for a subset of the entries inCware "masked" (zero-ed out) such that they do not enter into the optimization objective for latent factors. Exactly which entries are masked (zero-ed) depends on the masking strategy. Typically, we mask entries associated with FPs and FNs. We may also wish to mask entries with high uncertainty degrees (E.g., probability values close to 0.5 or other probability threshold, for which the labeling is unclear).
Related¶
Pc: Color matrix
In [ ]:
Copied!
import polarity_models as pm
"""
Setting up confidence Matrix.
Key Parameters
--------------
X: probabilty/rating matrix, X=[R|T]
L: labels (including the guesstimated labels associated with T)
p_threshold:
policy_threshold:
options: 'fmax', 'balanced', ...
this determines how `p_threshold` is estimated
conf_measure: Measure of reliabliity (of probability scores)
alpha: The factor by which confidence scores in C0 is to be scaled
in the latent-factor optimization objective
beta: The factor by which TP-specific confidence scores are amplified (
so that these terms are penalized more severely in the cost function,
if not approximated well)
"""
########################################################################
n_factors = 100
alpha = 100.0
beta = 1.0
conf_measure = 'brier' # 'brier', 'uniform', ...
policy_threshold = 'balanced'
Pc, C0, Cw, Cn, *rest = \
uc.evalConfidenceMatrices(X, L, alpha=alpha,
p_threshold=p_threshold,
conf_measure=conf_measure, policy_threshold=policy_threshold,
# Optional debug/test parameters
U=U, fold_number=fold_number,
# n_train = n_train,
is_cascade=True,
verbose=1)
y_colors = pm.verify_colors(Pc, X)
# Cn = uc.shift(Cn, -1.0)
############################################################
import polarity_models as pm
"""
Setting up confidence Matrix.
Key Parameters
--------------
X: probabilty/rating matrix, X=[R|T]
L: labels (including the guesstimated labels associated with T)
p_threshold:
policy_threshold:
options: 'fmax', 'balanced', ...
this determines how `p_threshold` is estimated
conf_measure: Measure of reliabliity (of probability scores)
alpha: The factor by which confidence scores in C0 is to be scaled
in the latent-factor optimization objective
beta: The factor by which TP-specific confidence scores are amplified (
so that these terms are penalized more severely in the cost function,
if not approximated well)
"""
########################################################################
n_factors = 100
alpha = 100.0
beta = 1.0
conf_measure = 'brier' # 'brier', 'uniform', ...
policy_threshold = 'balanced'
Pc, C0, Cw, Cn, *rest = \
uc.evalConfidenceMatrices(X, L, alpha=alpha,
p_threshold=p_threshold,
conf_measure=conf_measure, policy_threshold=policy_threshold,
# Optional debug/test parameters
U=U, fold_number=fold_number,
# n_train = n_train,
is_cascade=True,
verbose=1)
y_colors = pm.verify_colors(Pc, X)
# Cn = uc.shift(Cn, -1.0)
############################################################
================================================================================
(evalConfidenceMatrix) policy_filtering: item, policy_opt: rating | conf_measure: brier | policy_threshold: balanced, ratio_users: 0.5, ratio_small_class: 0, supervised? True, mask_all_test? False
================================================================================
... Filtering policy in training split: item =?= test split: polarity
################################################################################
(evalConfidenceMatrix) labeling_model: simple | constrained? True, stochastic? True, est sample type? False
################################################################################
... Balance class | balance sample size distribution? False, balance class conf scores? False
... Posthoc weight adjustments? | beta: 1.0, suppress_negative_examples: False
================================================================================
(toConfidenceMatrix) List of proba thresholds given policy: balanced | sorted according to -- classifer --
================================================================================
... [1] GNB: p_th = 0.234137
... [2] KNNC: p_th = 0.492823
... [3] MLPClassifier: p_th = 0.241718
... [4] QDA: p_th = 0.000486
... [5] RF: p_th = 0.500503
(toConfidenceMatrix) Computing conficence scores using conf_measure: brier
... p_threshold? True | message passing? False | policy: balanced
(confidence2D) item/data-wise confidence score distributions ...
... data [1879] | [0.2 0.8 0.8 0.8 0.4]
... data [711] | [0.2 0.8 0.8 0.8 0.4]
... data [3243] | [0.2 0.8 0.8 0.8 0.4]
... data [3651] | [0.2 0.8 0.8 0.8 0.4]
... data [3852] | [0.2 0.8 0.8 0.8 0.4]
(confidence2D) confidence weights (mode=brier)
... Wu:
[0.75 0.884 0.91 0.8 0.904]
... Wi:
[0.2 0.8 0.8 0.8 0.4 0.8 0.8 0.4 0.4 0.6]
... W:
[[0.15 0.6 0.6 0.6 0.3 0.6 0.6 0.3 0.3 0.45 ]
[0.177 0.707 0.707 0.707 0.354 0.707 0.707 0.354 0.354 0.53 ]
[0.182 0.728 0.728 0.728 0.364 0.728 0.728 0.364 0.364 0.546]
[0.16 0.64 0.64 0.64 0.32 0.64 0.64 0.32 0.32 0.48 ]
[0.181 0.723 0.723 0.723 0.362 0.723 0.723 0.362 0.362 0.543]]
... Colored polarity matrix (Pc) | n_negative: 5792, n_neutral: 0 n_positive: 19208 | data: training set
[verify] Cui, Pc converted to sparse matrix.
(toConfidenceMatrix) Cui: hape(Cui)=(5, 5000), n_zeros (uncertain)=0 (>? 0) vs n_nonzeros=25000 (masked ratio=0.0)
(balance_and_scale) Balancing class weights by considering size disparity ...
================================================================================
(verify_confidence_matrix) Are the confidence scores taking on values as expected?
================================================================================
--------------------------------------------------------------------------------
(before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set
--------------------------------------------------------------------------------
[verify] n(TP+TN): 19208, n(FP+FN): 5792, ratio: 0.76832
-- Class-wise weight distributions --
... Class Positive (+): min = 0.14999366438129264, max = 0.9098540358108046, mean = 0.45297333288677954, median = 0.4798515549692781
... Class Negative (-): min = 0.14999366438129264, max = 0.9098540358108046, mean = 0.726047247199113, median = 0.7278832286486437
--- Confidence score (weight) sum total per class ---
... N(TP): 717, N(TN): 18491, N(TP)/N(TN): 0.038775620572170245
... W(TP): 324.7818796798209, W(TN): 13425.339647958797, W(TP)/W(TN): 0.02419170674234682
... Balanced? W(TP)/W(TN)=0.02419170674234682 ~? 1.0
================================================================================
(verify_confidence_matrix) Have we masked the neutral (uncertain) and negative entries (FPs, FNs)?
================================================================================
[verify] Neutral and negatives (FP, FN) are masked: n(zeros): 505 >=? n(fp+fn): 5792
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[0.53 0.53 0.354 0.354 0.354 0.884 0.707 0.53 0.53 0.354]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.5302648591202681
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[0.884 0.707 0.884 0.884 0.707 0.53 0.884 0.707 0.884 0.53 ]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.7070198121603575
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[0.53 0.53 0.354 0.354 0.354 0.884 0.707 0.53 0.53 0.354]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.5302648591202681
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[0.884 0.707 0.884 0.884 0.707 0.53 0.884 0.707 0.884 0.53 ]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.7070198121603575
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[0.32 0.64 0.48 0.32 0.32 0.48 0.16 0.64 0.48 0.32]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.3199010366461854
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[0.48 0.64 0.64 0.64 0.8 0.64 0.8 0.64 0.8 0.8 ]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.6398020732923708
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[0.53 0.53 0.354 0.354 0.354 0.884 0.707 0.53 0.53 0.354]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.5302648591202681
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[0.884 0.707 0.884 0.884 0.707 0.53 0.884 0.707 0.884 0.53 ]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.7070198121603575
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[0.32 0.64 0.48 0.32 0.32 0.48 0.16 0.64 0.48 0.32]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.3199010366461854
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[0.48 0.64 0.64 0.64 0.8 0.64 0.8 0.64 0.8 0.8 ]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.6398020732923708
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[MLPClassifier]: n(P): 450, n(TP): 104, recall: 0.2311111111111111
[0.543 0.723 0.723 0.723 0.723 0.723 0.723 0.543 0.723 0.723]
... Min: 0.18085353425941597, max: 0.9042676712970797, median: 0.5425606027782478
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[MLPClassifier]: n(N): 4550, n(TN): 4106, TNR: 0.9024175824175824
[0.904 0.723 0.904 0.723 0.723 0.904 0.904 0.543 0.543 0.723]
... Min: 0.18085353425941597, max: 0.9042676712970797, median: 0.7234141370376639
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
--------------------------------------------------------------------------------
[info] Before re-weighting TP(+) | 5 numbers: (0.14999366438129264, 0.29998732876258527, 0.4798515549692781, 0.6398020732923708, 0.9098540358108046)
TN(-) | 5 numbers: (0.14999366438129264, 0.6398020732923708, 0.7278832286486437, 0.8837747652004468, 0.9098540358108046)
FP(-) | 5 numbers: (0.0, 0.44998099314387785, 0.5459124214864828, 0.6398020732923708, 0.7278832286486437)
FN(+) | 5 numbers: (0.0, 0.0, 0.17675495304008937, 0.3199010366461854, 0.7278832286486437)
... Reweight C inversely proprotional to samples sizes >
... N (total): 25000... wtp: 0.5970736096242829, wtn: 0.023151899740447287, wfp: 0.10051697067401053, wfn: 0.27925751996125947
(balance_and_scale) Discount test sample weights by 0.5
[info] After re-weighting TP(+) | 5 numbers: (0.08955725861291162, 0.17911451722582325, 0.28650670000933187, 0.38200893334577585, 0.5432498333927787)
TN(-) | 5 numbers: (0.0034726382794579864, 0.014812633454595277, 0.016851879532426468, 0.020461064757058087, 0.021064849415533084)
FP(-) | 5 numbers: (0.0, 0.0452307262917053, 0.05487346286113486, 0.06431096623830038, 0.07316461714817982)
FN(+) | 5 numbers: (0.0, 0.0, 0.04936014982684424, 0.08933477012684972, 0.20326686525381463)
================================================================================
(verify_confidence_matrix) Are the confidence scores taking on values as expected?
================================================================================
--------------------------------------------------------------------------------
(after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set
--------------------------------------------------------------------------------
[verify] n(TP+TN): 19208, n(FP+FN): 5792, ratio: 0.76832
-- Class-wise weight distributions --
... Class Positive (+): min = 8.955725861291162, max = 54.324983339277864, mean = 27.04584229302513, median = 28.650670000933186
... Class Negative (-): min = 0.34726382794579863, max = 2.1064849415533082, mean = 1.680671393137568, median = 1.6851879532426468
--- Confidence score (weight) sum total per class ---
... N(TP): 717, N(TN): 18491, N(TP)/N(TN): 0.038775620572170245
... W(TP): 19391.86892409902, W(TN): 31077.294730506772, W(TP)/W(TN): 0.6239883198412104
... Balanced? W(TP)/W(TN)=0.6239883198412104 ~? 1.0
================================================================================
(verify_confidence_matrix) Have we masked the neutral (uncertain) and negative entries (FPs, FNs)?
================================================================================
[verify] Neutral and negatives (FP, FN) are masked: n(zeros): 505 >=? n(fp+fn): 5792
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[GNB]: n(P): 450, n(TP): 123, recall: 0.2733333333333333
[32.595 43.46 43.46 54.325 43.46 32.595 43.46 32.595 32.595 43.46 ]
... Min: 10.864996667855573, max: 54.324983339277864, median: 32.59499000356672
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[GNB]: n(N): 4550, n(TN): 4013, TNR: 0.881978021978022
[2.106 1.685 1.685 1.685 2.106 2.106 2.106 1.264 1.685 2.106]
... Min: 0.4212969883106617, max: 2.1064849415533082, median: 1.6851879532426468
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[GNB]: n(P): 450, n(TP): 123, recall: 0.2733333333333333
[32.595 43.46 43.46 54.325 43.46 32.595 43.46 32.595 32.595 43.46 ]
... Min: 10.864996667855573, max: 54.324983339277864, median: 32.59499000356672
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[GNB]: n(N): 4550, n(TN): 4013, TNR: 0.881978021978022
[2.106 1.685 1.685 1.685 2.106 2.106 2.106 1.264 1.685 2.106]
... Min: 0.4212969883106617, max: 2.1064849415533082, median: 1.6851879532426468
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[38.201 28.651 19.1 19.1 28.651 9.55 38.201 28.651 19.1 28.651]
... Min: 9.550223333644396, max: 47.75111666822198, median: 19.100446667288793
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[1.111 1.481 1.481 1.481 1.852 1.481 1.852 1.481 1.852 1.852]
... Min: 0.3703158363648819, max: 1.8515791818244094, median: 1.4812633454595276
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[GNB]: n(P): 450, n(TP): 123, recall: 0.2733333333333333
[32.595 43.46 43.46 54.325 43.46 32.595 43.46 32.595 32.595 43.46 ]
... Min: 10.864996667855573, max: 54.324983339277864, median: 32.59499000356672
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[GNB]: n(N): 4550, n(TN): 4013, TNR: 0.881978021978022
[2.106 1.685 1.685 1.685 2.106 2.106 2.106 1.264 1.685 2.106]
... Min: 0.4212969883106617, max: 2.1064849415533082, median: 1.6851879532426468
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[38.201 28.651 19.1 19.1 28.651 9.55 38.201 28.651 19.1 28.651]
... Min: 9.550223333644396, max: 47.75111666822198, median: 19.100446667288793
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[1.111 1.481 1.481 1.481 1.852 1.481 1.852 1.481 1.852 1.852]
... Min: 0.3703158363648819, max: 1.8515791818244094, median: 1.4812633454595276
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[MLPClassifier]: n(P): 450, n(TP): 104, recall: 0.2311111111111111
[32.395 43.193 43.193 43.193 43.193 43.193 43.193 32.395 43.193 43.193]
... Min: 10.798287251357841, max: 53.99143625678919, median: 32.39486175407352
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[MLPClassifier]: n(N): 4550, n(TN): 4106, TNR: 0.9024175824175824
[2.094 1.675 2.094 1.675 1.675 2.094 2.094 1.256 1.256 1.675]
... Min: 0.41871028928795473, max: 2.0935514464397733, median: 1.674841157151819
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[info] class stats: {'n_pos': 450, 'n_neg': 4550, 'n_min_class': 450, 'n_max_class': 4550, 'n': 5000, 'r_pos': 0.09, 1: 0.09, 'r_neg': 0.91, 0: 0.91, 'r_min': 0.09, 'r_minority': 0.09, 'r_max': 0.91, 'r_majority': 0.91, 'min_class': 1, 'minority_class': 1, 'max_class': 0, 'majority_class': 0, 'r_max_to_min': 10.11111111111111, 'multiple': 10.11111111111111, 'r_min_to_max': 0.0989010989010989, 'is_balanced': False}
(make_cn) Using WEIGHTED confidence matrix to approximate ratings ...
(balance_and_scale) Balancing class weights by considering size disparity ...
================================================================================
(verify_confidence_matrix) Are the confidence scores taking on values as expected?
================================================================================
--------------------------------------------------------------------------------
(before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set
--------------------------------------------------------------------------------
[verify] n(TP+TN): 19208, n(FP+FN): 5792, ratio: 0.76832
-- Class-wise weight distributions --
... Class Positive (+): min = 0.14999366438129264, max = 0.9098540358108046, mean = 0.45297333288677954, median = 0.4798515549692781
... Class Negative (-): min = 0.14999366438129264, max = 0.9098540358108046, mean = 0.726047247199113, median = 0.7278832286486437
--- Confidence score (weight) sum total per class ---
... N(TP): 717, N(TN): 18491, N(TP)/N(TN): 0.038775620572170245
... W(TP): 324.7818796798209, W(TN): 13425.339647958797, W(TP)/W(TN): 0.02419170674234682
... Balanced? W(TP)/W(TN)=0.02419170674234682 ~? 1.0
================================================================================
(verify_confidence_matrix) Have we masked the neutral (uncertain) and negative entries (FPs, FNs)?
================================================================================
[verify] Neutral and negatives (FP, FN) are masked: n(zeros): 5792 >=? n(fp+fn): 5792
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[0.53 0.53 0.354 0.354 0.354 0.884 0.707 0.53 0.53 0.354]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.5302648591202681
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[0.884 0.707 0.884 0.884 0.707 0.53 0.884 0.707 0.884 0.53 ]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.7070198121603575
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[0.53 0.53 0.354 0.354 0.354 0.884 0.707 0.53 0.53 0.354]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.5302648591202681
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[0.884 0.707 0.884 0.884 0.707 0.53 0.884 0.707 0.884 0.53 ]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.7070198121603575
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[0.32 0.64 0.48 0.32 0.32 0.48 0.16 0.64 0.48 0.32]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.3199010366461854
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[0.48 0.64 0.64 0.64 0.8 0.64 0.8 0.64 0.8 0.8 ]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.6398020732923708
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[0.15 0.15 0.45 0.3 0.3 0.3 0.15 0.45 0.3 0.3 ]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.29998732876258527
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[0.6 0.75 0.75 0.75 0.3 0.6 0.6 0.3 0.75 0.75]
... Min: 0.14999366438129264, max: 0.7499683219064631, median: 0.5999746575251705
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[0.53 0.53 0.354 0.354 0.354 0.884 0.707 0.53 0.53 0.354]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.5302648591202681
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[0.884 0.707 0.884 0.884 0.707 0.53 0.884 0.707 0.884 0.53 ]
... Min: 0.17675495304008937, max: 0.8837747652004468, median: 0.7070198121603575
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[0.32 0.64 0.48 0.32 0.32 0.48 0.16 0.64 0.48 0.32]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.3199010366461854
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[0.48 0.64 0.64 0.64 0.8 0.64 0.8 0.64 0.8 0.8 ]
... Min: 0.1599505183230927, max: 0.7997525916154635, median: 0.6398020732923708
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[MLPClassifier]: n(P): 450, n(TP): 104, recall: 0.2311111111111111
[0.543 0.723 0.723 0.723 0.723 0.723 0.723 0.543 0.723 0.723]
... Min: 0.18085353425941597, max: 0.9042676712970797, median: 0.5425606027782478
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[MLPClassifier]: n(N): 4550, n(TN): 4106, TNR: 0.9024175824175824
[0.904 0.723 0.904 0.723 0.723 0.904 0.904 0.543 0.543 0.723]
... Min: 0.18085353425941597, max: 0.9042676712970797, median: 0.7234141370376639
... condition: ((before) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
--------------------------------------------------------------------------------
[info] Before re-weighting TP(+) | 5 numbers: (0.14999366438129264, 0.29998732876258527, 0.4798515549692781, 0.6398020732923708, 0.9098540358108046)
TN(-) | 5 numbers: (0.14999366438129264, 0.6398020732923708, 0.7278832286486437, 0.8837747652004468, 0.9098540358108046)
FP(-) | 5 numbers: (0.0, 0.0, 0.0, 0.0, 0.0)
FN(+) | 5 numbers: (0.0, 0.0, 0.0, 0.0, 0.0)
... Reweight C inversely proprotional to samples sizes >
... N (total): 25000... wtp: 0.5970736096242829, wtn: 0.023151899740447287, wfp: 0.10051697067401053, wfn: 0.27925751996125947
(balance_and_scale) Discount test sample weights by 0.5
[info] After re-weighting TP(+) | 5 numbers: (0.08955725861291162, 0.17911451722582325, 0.28650670000933187, 0.38200893334577585, 0.5432498333927787)
TN(-) | 5 numbers: (0.0034726382794579864, 0.014812633454595277, 0.016851879532426468, 0.020461064757058087, 0.021064849415533084)
FP(-) | 5 numbers: (0.0, 0.0, 0.0, 0.0, 0.0)
FN(+) | 5 numbers: (0.0, 0.0, 0.0, 0.0, 0.0)
================================================================================
(verify_confidence_matrix) Are the confidence scores taking on values as expected?
================================================================================
--------------------------------------------------------------------------------
(after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set
--------------------------------------------------------------------------------
[verify] n(TP+TN): 19208, n(FP+FN): 5792, ratio: 0.76832
-- Class-wise weight distributions --
... Class Positive (+): min = 8.955725861291162, max = 54.324983339277864, mean = 27.04584229302513, median = 28.650670000933186
... Class Negative (-): min = 0.34726382794579863, max = 2.1064849415533082, mean = 1.680671393137568, median = 1.6851879532426468
--- Confidence score (weight) sum total per class ---
... N(TP): 717, N(TN): 18491, N(TP)/N(TN): 0.038775620572170245
... W(TP): 19391.86892409902, W(TN): 31077.294730506772, W(TP)/W(TN): 0.6239883198412104
... Balanced? W(TP)/W(TN)=0.6239883198412104 ~? 1.0
================================================================================
(verify_confidence_matrix) Have we masked the neutral (uncertain) and negative entries (FPs, FNs)?
================================================================================
[verify] Neutral and negatives (FP, FN) are masked: n(zeros): 5792 >=? n(fp+fn): 5792
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[GNB]: n(P): 450, n(TP): 123, recall: 0.2733333333333333
[32.595 43.46 43.46 54.325 43.46 32.595 43.46 32.595 32.595 43.46 ]
... Min: 10.864996667855573, max: 54.324983339277864, median: 32.59499000356672
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[GNB]: n(N): 4550, n(TN): 4013, TNR: 0.881978021978022
[2.106 1.685 1.685 1.685 2.106 2.106 2.106 1.264 1.685 2.106]
... Min: 0.4212969883106617, max: 2.1064849415533082, median: 1.6851879532426468
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[GNB]: n(P): 450, n(TP): 123, recall: 0.2733333333333333
[32.595 43.46 43.46 54.325 43.46 32.595 43.46 32.595 32.595 43.46 ]
... Min: 10.864996667855573, max: 54.324983339277864, median: 32.59499000356672
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[GNB]: n(N): 4550, n(TN): 4013, TNR: 0.881978021978022
[2.106 1.685 1.685 1.685 2.106 2.106 2.106 1.264 1.685 2.106]
... Min: 0.4212969883106617, max: 2.1064849415533082, median: 1.6851879532426468
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[38.201 28.651 19.1 19.1 28.651 9.55 38.201 28.651 19.1 28.651]
... Min: 9.550223333644396, max: 47.75111666822198, median: 19.100446667288793
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[1.111 1.481 1.481 1.481 1.852 1.481 1.852 1.481 1.852 1.852]
... Min: 0.3703158363648819, max: 1.8515791818244094, median: 1.4812633454595276
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
-- Base predictor weight distributions --
[verify] [TP] BP=[RF]: n(P): 450, n(TP): 161, recall: 0.35777777777777775
[17.911 17.911 17.911 17.911 8.956 17.911 17.911 17.911 8.956 8.956]
... Min: 8.955725861291162, max: 44.778629306455805, median: 17.911451722582324
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[RF]: n(N): 4550, n(TN): 3211, TNR: 0.7057142857142857
[1.736 1.736 1.389 1.736 1.736 1.736 1.389 1.736 1.736 1.389]
... Min: 0.34726382794579863, max: 1.736319139728993, median: 1.3890553117831945
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[KNNC]: n(P): 450, n(TP): 128, recall: 0.28444444444444444
[31.661 21.107 21.107 10.554 21.107 42.214 31.661 42.214 42.214 42.214]
... Min: 10.553571783061678, max: 52.76785891530839, median: 31.660715349185033
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[KNNC]: n(N): 4550, n(TN): 3943, TNR: 0.8665934065934066
[2.046 1.637 1.228 1.228 1.228 1.637 0.818 2.046 2.046 1.637]
... Min: 0.40922129514116173, max: 2.046106475705809, median: 1.636885180564647
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[GNB]: n(P): 450, n(TP): 123, recall: 0.2733333333333333
[32.595 43.46 43.46 54.325 43.46 32.595 43.46 32.595 32.595 43.46 ]
... Min: 10.864996667855573, max: 54.324983339277864, median: 32.59499000356672
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[GNB]: n(N): 4550, n(TN): 4013, TNR: 0.881978021978022
[2.106 1.685 1.685 1.685 2.106 2.106 2.106 1.264 1.685 2.106]
... Min: 0.4212969883106617, max: 2.1064849415533082, median: 1.6851879532426468
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[QDA]: n(P): 450, n(TP): 201, recall: 0.44666666666666666
[38.201 28.651 19.1 19.1 28.651 9.55 38.201 28.651 19.1 28.651]
... Min: 9.550223333644396, max: 47.75111666822198, median: 19.100446667288793
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[QDA]: n(N): 4550, n(TN): 3218, TNR: 0.7072527472527472
[1.111 1.481 1.481 1.481 1.852 1.481 1.852 1.481 1.852 1.852]
... Min: 0.3703158363648819, max: 1.8515791818244094, median: 1.4812633454595276
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TP] BP=[MLPClassifier]: n(P): 450, n(TP): 104, recall: 0.2311111111111111
[32.395 43.193 43.193 43.193 43.193 43.193 43.193 32.395 43.193 43.193]
... Min: 10.798287251357841, max: 53.99143625678919, median: 32.39486175407352
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[verify] [TN] BP=[MLPClassifier]: n(N): 4550, n(TN): 4106, TNR: 0.9024175824175824
[2.094 1.675 2.094 1.675 1.675 2.094 2.094 1.256 1.256 1.675]
... Min: 0.41871028928795473, max: 2.0935514464397733, median: 1.674841157151819
... condition: ((after) balanced + magnified (alpha=100.0, beta=1.0) | dtype: training set)
[info] class stats: {'n_pos': 450, 'n_neg': 4550, 'n_min_class': 450, 'n_max_class': 4550, 'n': 5000, 'r_pos': 0.09, 1: 0.09, 'r_neg': 0.91, 0: 0.91, 'r_min': 0.09, 'r_minority': 0.09, 'r_max': 0.91, 'r_majority': 0.91, 'min_class': 1, 'minority_class': 1, 'max_class': 0, 'majority_class': 0, 'r_max_to_min': 10.11111111111111, 'multiple': 10.11111111111111, 'r_min_to_max': 0.0989010989010989, 'is_balanced': False}
In [ ]:
Copied!
from numpy import linalg as LA
from scipy.spatial import distance
from sklearn.metrics import f1_score
from analyzer import analyze_matrix_type
import utils_cf as uc
import cf
fold_number = 0
# Uncomment the following if starting the notebook from here (bypassing training and optimization)
###############################################################################
# ret = ustk.get_pretrained_model_with_confidence_matrix(input_dir,
# base_learners,
# fold_number,
# policy_threshold='balanced',
# conf_measure='brier',
# verbose=1)
# X, L, U, p_threshold, n_train = ret['X'], ret['L'], ret['U'], ret['p_threshold'], ret['n_train']
# R, T = X[:,:n_train], X[:,n_train:]
# Cn, Pc = ret['Cn'], ret['Pc']
###############################################################################
ustk.verify_shape(X, R, T, L, U, p_threshold) # verify the shape of all key quantities
# Check: Wherever Pc is negative, the corresponding entries in Cn must be 0 (By constrast, C is a full/dense confidence matrix)
assert np.all(Cn[Pc < 0]==0)
assert np.all(Cn[Pc > 0]>0)
from numpy import linalg as LA
from scipy.spatial import distance
from sklearn.metrics import f1_score
from analyzer import analyze_matrix_type
import utils_cf as uc
import cf
fold_number = 0
# Uncomment the following if starting the notebook from here (bypassing training and optimization)
###############################################################################
# ret = ustk.get_pretrained_model_with_confidence_matrix(input_dir,
# base_learners,
# fold_number,
# policy_threshold='balanced',
# conf_measure='brier',
# verbose=1)
# X, L, U, p_threshold, n_train = ret['X'], ret['L'], ret['U'], ret['p_threshold'], ret['n_train']
# R, T = X[:,:n_train], X[:,n_train:]
# Cn, Pc = ret['Cn'], ret['Pc']
###############################################################################
ustk.verify_shape(X, R, T, L, U, p_threshold) # verify the shape of all key quantities
# Check: Wherever Pc is negative, the corresponding entries in Cn must be 0 (By constrast, C is a full/dense confidence matrix)
assert np.all(Cn[Pc < 0]==0)
assert np.all(Cn[Pc > 0]>0)
Sanity check on the particle types ...
In [ ]:
Copied!
import collections
from analyzer import is_sparse
print(f"> Is Pc sparse? {is_sparse(Pc)}")
print(f"> Is Cn sparse? {is_sparse(Cn)}")
print(np.unique(Pc.A if is_sparse(Pc) else Pc))
assert (X.shape == C0.shape ) and (Pc.shape == Cn.shape)
counter = collections.Counter((Pc.A if is_sparse(Pc) else Pc).reshape(-1, ))
print(counter)
import collections
from analyzer import is_sparse
print(f"> Is Pc sparse? {is_sparse(Pc)}")
print(f"> Is Cn sparse? {is_sparse(Cn)}")
print(np.unique(Pc.A if is_sparse(Pc) else Pc))
assert (X.shape == C0.shape ) and (Pc.shape == Cn.shape)
counter = collections.Counter((Pc.A if is_sparse(Pc) else Pc).reshape(-1, ))
print(counter)
> Is Pc sparse? True
> Is Cn sparse? True
[-2. -1. 1. 2.]
Counter({1.0: 18491, -2.0: 4259, -1.0: 1533, 2.0: 717})
Experiment #1: CFNet using MSE loss¶
1A. Preparing training data for CFNet using MSE loss¶
We'll begin by experimenting on simple, frequently-used loss functions:
- If we intend to approximate probability scores in X, then MSE would be a reasonable loss function to try. Specifically, we want use the latent factors to approximate the probabilities in X. CFNet essentially learns a function f(x, y) that approximates the probability score associated with user x and item y.
- If we intend to approximate the labeling entailed by X (given base predictor's probability thresholds,
p_threshold), then BCE loss would be a reasonable choice.
In this first experiment, we will use latent factors to approximate probability scores in
Xdirectly
In [ ]:
Copied!
import data_pipeline as dp
# from utils_cf import estimateLabelMatrix, estimateLabels
df = dp.rating_matrix_to_dataframe(X, shuffle=True)
n_users0, n_items0 = X.shape
# [test]
#########################################
assert df.shape[0] == X.size
# df.astype({'user': 'int32', 'item': 'int64'})
for i, dfi in df.iterrows():
if i > 100: break
r, c, score = dfi['user'], dfi['item'], dfi['rating']
assert np.allclose(X[r][c], score)
#########################################
C = Cn # use the masked and rescaled confidence matrix
P = uc.to_preference(Pc) # color matrix to probability filter (where {TP, TN} maps to 1 and {FP, FN} maps to 0)
Xc, yc, weights, proba_filter = dp.matrix_to_augmented_training_data(X, C, P) # NOTE: Don't overwrite X (`Xc` is not the same as `X`, which is a rating matrix)
# Sample weights
##########################################
# Without amplifying TP's weights (TP and TN are equally important)
# sample_weights = weights * proba_filter
# Amplifying TP weights inversely proportioinal to the TP sample size relative to TN
# E.g. If sample size of TP is 1/10th that of TN, then TP's weight should be 10-fold larger
sample_weights = dp.unravel(C, normalize=False) # Cn is a masked and balanced version of C0
assert len(sample_weights) == Xc.shape[0]
##########################################
test_size = 0.1
# Make train-test split with sample weights
split_pt = int((1-test_size) * Xc.shape[0])
X_train, X_val, y_train, y_val, W_train, W_val = (
Xc[:split_pt],
Xc[split_pt:],
yc[:split_pt],
yc[split_pt:],
sample_weights[:split_pt],
sample_weights[split_pt:]
)
# A few other ways of making train-test splits
# Create a training dataset that includes sample weights
# train_dataset = tf.data.Dataset.from_tensor_slices((Xc, yc, sample_weights))
# splits = dp.get_dataset_partitions_tf(train_dataset, Xc.shape[0],
# train_split=0.8, val_split=0.0, test_split=0.2,
# shuffle=True, shuffle_size=10000, random_state=53)
# train_dataset, _, test_dataset = splits
# X_train, X_val, y_train, y_val = train_test_split(Xc, yc, test_size=test_size, random_state=53)
# Test
print(f"> shape(X_train): {X_train.shape}, shape(X_val): {X_val.shape}")
assert X.size == X_train.shape[0] + X_val.shape[0]
import data_pipeline as dp
# from utils_cf import estimateLabelMatrix, estimateLabels
df = dp.rating_matrix_to_dataframe(X, shuffle=True)
n_users0, n_items0 = X.shape
# [test]
#########################################
assert df.shape[0] == X.size
# df.astype({'user': 'int32', 'item': 'int64'})
for i, dfi in df.iterrows():
if i > 100: break
r, c, score = dfi['user'], dfi['item'], dfi['rating']
assert np.allclose(X[r][c], score)
#########################################
C = Cn # use the masked and rescaled confidence matrix
P = uc.to_preference(Pc) # color matrix to probability filter (where {TP, TN} maps to 1 and {FP, FN} maps to 0)
Xc, yc, weights, proba_filter = dp.matrix_to_augmented_training_data(X, C, P) # NOTE: Don't overwrite X (`Xc` is not the same as `X`, which is a rating matrix)
# Sample weights
##########################################
# Without amplifying TP's weights (TP and TN are equally important)
# sample_weights = weights * proba_filter
# Amplifying TP weights inversely proportioinal to the TP sample size relative to TN
# E.g. If sample size of TP is 1/10th that of TN, then TP's weight should be 10-fold larger
sample_weights = dp.unravel(C, normalize=False) # Cn is a masked and balanced version of C0
assert len(sample_weights) == Xc.shape[0]
##########################################
test_size = 0.1
# Make train-test split with sample weights
split_pt = int((1-test_size) * Xc.shape[0])
X_train, X_val, y_train, y_val, W_train, W_val = (
Xc[:split_pt],
Xc[split_pt:],
yc[:split_pt],
yc[split_pt:],
sample_weights[:split_pt],
sample_weights[split_pt:]
)
# A few other ways of making train-test splits
# Create a training dataset that includes sample weights
# train_dataset = tf.data.Dataset.from_tensor_slices((Xc, yc, sample_weights))
# splits = dp.get_dataset_partitions_tf(train_dataset, Xc.shape[0],
# train_split=0.8, val_split=0.0, test_split=0.2,
# shuffle=True, shuffle_size=10000, random_state=53)
# train_dataset, _, test_dataset = splits
# X_train, X_val, y_train, y_val = train_test_split(Xc, yc, test_size=test_size, random_state=53)
# Test
print(f"> shape(X_train): {X_train.shape}, shape(X_val): {X_val.shape}")
assert X.size == X_train.shape[0] + X_val.shape[0]
> shape(X_train): (22500, 2), shape(X_val): (2500, 2)
More complex loss functions¶
- We could further define a confidence-weighted loss fucntion that is also aware of the "color" as given by
confidence_weighted_loss(...)below. However, this initial implementation won't work with TF's model.fit(). Why?- The model.fit() framework only accepts loss functions that take on (y_true, y_pred) as inputs; if we need to pass on additional parameters that also influence the loss calculation, we need to do additoinal work ...
if-then-elsestatements won't work and are not recommended.- We also need to ensure that boolean masks are converted to tf.float32 so that they can be used to establish the weighted loss
- Given the observations above, an example of the confidence weighted_loss is implemented in
cf_modelsmodule, which will be introduced shortly.
In [ ]:
Copied!
# y_pred: f( <x, y> ) # <x, y> denotes the dot product between x and y
def confidence_weighted_loss(y_true, y_pred, weights=None, colors=None): # this has to be used with .add_loss() with greater flexibility
if weights is None:
weights = K.ones_like(y_pred)
if colors is None:
# confidence-weighted sum of squares
wmse = weights * K.square(y_pred - y_true, axis=-1) # difference between predicted and "true" probability
else:
# condition
mask_tp = K.equal(colors, 2)
mask_tn = K.equal(colors, 1)
mask_fp = K.equal(colors, -2)
mask_fn = K.equal(colors, -1)
# if TP, want y_pred >= y_true, i.e. the larger (the closer to 1), the better
loss_tp = weights * K.square(K.maximum(y_true-y_pred, 0)) # if y_pred > y_true => y_true-y_pred < 0 => no loss, if ow, then the smaller, the higher the penalty (quadratically)
# if TN, want y_pred < y_true, i.e. the smaller (the closer to 0), the better
loss_tn = weights * K.square(K.maximum(y_pred-y_true, 0)) # if y_true>y_pred => y_pred-y_true < 0 => no loss
# if FP, y_pred must've been too large, want y_pred smaller
loss_fp = loss_tn
# loss_fp = weights * K.pow(K.maximum(y_true-y_pred, 0) # may need to be 'a lot' smaller => could penalize error cubically instead
# if FN, y_pred must've been too small, want y_pred larger
loss_fn = loss_tp
# loss_fn = weights * K.pow(K.maximum(y_pred-y_true, 0), 3) # penalize cubically or any exponent > 2
wmse = mask_tp * loss_tp + mask_tn * loss_tn + mask_fp * loss_fp + mask_fn * loss_fn
return wmse
# y_pred: f( ) # denotes the dot product between x and y
def confidence_weighted_loss(y_true, y_pred, weights=None, colors=None): # this has to be used with .add_loss() with greater flexibility
if weights is None:
weights = K.ones_like(y_pred)
if colors is None:
# confidence-weighted sum of squares
wmse = weights * K.square(y_pred - y_true, axis=-1) # difference between predicted and "true" probability
else:
# condition
mask_tp = K.equal(colors, 2)
mask_tn = K.equal(colors, 1)
mask_fp = K.equal(colors, -2)
mask_fn = K.equal(colors, -1)
# if TP, want y_pred >= y_true, i.e. the larger (the closer to 1), the better
loss_tp = weights * K.square(K.maximum(y_true-y_pred, 0)) # if y_pred > y_true => y_true-y_pred < 0 => no loss, if ow, then the smaller, the higher the penalty (quadratically)
# if TN, want y_pred < y_true, i.e. the smaller (the closer to 0), the better
loss_tn = weights * K.square(K.maximum(y_pred-y_true, 0)) # if y_true>y_pred => y_pred-y_true < 0 => no loss
# if FP, y_pred must've been too large, want y_pred smaller
loss_fp = loss_tn
# loss_fp = weights * K.pow(K.maximum(y_true-y_pred, 0) # may need to be 'a lot' smaller => could penalize error cubically instead
# if FN, y_pred must've been too small, want y_pred larger
loss_fn = loss_tp
# loss_fn = weights * K.pow(K.maximum(y_pred-y_true, 0), 3) # penalize cubically or any exponent > 2
wmse = mask_tp * loss_tp + mask_tn * loss_tn + mask_fp * loss_fp + mask_fn * loss_fn
return wmse
1B. Create the model¶
- See
cf_modelsfor a few example models
In [ ]:
Copied!
class CFNet(keras.Model):
def __init__(self, n_users, n_items, embedding_size, **kwargs):
super(CFNet, self).__init__(**kwargs)
self.n_users = n_users
self.n_items = n_items
self.embedding_size = embedding_size
self.user_embedding = Embedding(
n_users,
embedding_size,
embeddings_initializer="he_normal",
embeddings_regularizer=keras.regularizers.l2(0.01),
)
self.user_bias = Embedding(n_users, 1)
self.item_embedding = Embedding(
n_items,
embedding_size,
embeddings_initializer="he_normal",
embeddings_regularizer=keras.regularizers.l2(0.01),
)
self.item_bias = Embedding(n_items, 1)
def call(self, inputs):
user_vector = self.user_embedding(inputs[:, 0])
user_bias = self.user_bias(inputs[:, 0])
item_vector = self.item_embedding(inputs[:, 1])
item_bias = self.item_bias(inputs[:, 1])
dot_user_item = tf.tensordot(user_vector, item_vector, 2)
# Add all the components (including bias)
x = dot_user_item + user_bias + item_bias
# The sigmoid activation forces the rating to between 0 and 1
return tf.nn.sigmoid(x)
class CFNet(keras.Model):
def __init__(self, n_users, n_items, embedding_size, **kwargs):
super(CFNet, self).__init__(**kwargs)
self.n_users = n_users
self.n_items = n_items
self.embedding_size = embedding_size
self.user_embedding = Embedding(
n_users,
embedding_size,
embeddings_initializer="he_normal",
embeddings_regularizer=keras.regularizers.l2(0.01),
)
self.user_bias = Embedding(n_users, 1)
self.item_embedding = Embedding(
n_items,
embedding_size,
embeddings_initializer="he_normal",
embeddings_regularizer=keras.regularizers.l2(0.01),
)
self.item_bias = Embedding(n_items, 1)
def call(self, inputs):
user_vector = self.user_embedding(inputs[:, 0])
user_bias = self.user_bias(inputs[:, 0])
item_vector = self.item_embedding(inputs[:, 1])
item_bias = self.item_bias(inputs[:, 1])
dot_user_item = tf.tensordot(user_vector, item_vector, 2)
# Add all the components (including bias)
x = dot_user_item + user_bias + item_bias
# The sigmoid activation forces the rating to between 0 and 1
return tf.nn.sigmoid(x)
In [ ]:
Copied!
n_users = len(np.unique(df['user']))
n_items = len(np.unique(df['item']))
assert n_users == n_users0 == X.shape[0], f"n_users: {n_users} != n(rows) in X: {X.shape[0]}"
assert n_items == n_items0 == X.shape[1], f"n_items: {n_items} != n(cols) in X: {X.shape[1]}"
print(f'[info] shape(X): {X.shape} => n_users={n_users}, n_items={n_items}')
n_users = len(np.unique(df['user']))
n_items = len(np.unique(df['item']))
assert n_users == n_users0 == X.shape[0], f"n_users: {n_users} != n(rows) in X: {X.shape[0]}"
assert n_items == n_items0 == X.shape[1], f"n_items: {n_items} != n(cols) in X: {X.shape[1]}"
print(f'[info] shape(X): {X.shape} => n_users={n_users}, n_items={n_items}')
[info] shape(X): (5, 5000) => n_users=5, n_items=5000
In [ ]:
Copied!
n_factors = 100
epochs = 120
#######################################################
loss_fn = mse = tf.keras.losses.MeanSquaredError()
# Options:
# bce = tf.keras.losses.BinaryCrossentropy() # Note: Use BCE loss if we are approximating labels (0s and 1s)
# mse = tf.keras.losses.MeanSquaredError() # Note: Use MSE loss if we are approximating probability scores instead of labels
#######################################################
model = CFNet(n_users, n_items, n_factors)
model.compile(
loss=loss_fn, optimizer=keras.optimizers.Adam(lr=0.001)
)
n_factors = 100
epochs = 120
#######################################################
loss_fn = mse = tf.keras.losses.MeanSquaredError()
# Options:
# bce = tf.keras.losses.BinaryCrossentropy() # Note: Use BCE loss if we are approximating labels (0s and 1s)
# mse = tf.keras.losses.MeanSquaredError() # Note: Use MSE loss if we are approximating probability scores instead of labels
#######################################################
model = CFNet(n_users, n_items, n_factors)
model.compile(
loss=loss_fn, optimizer=keras.optimizers.Adam(lr=0.001)
)
1C. Train and evaluate the model¶
In [ ]:
Copied!
# Use TensorBoard to visualize metrics including loss and accuracy.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
history = model.fit(
x=X_train,
y=y_train,
sample_weight=W_train,
batch_size=64,
epochs=epochs,
verbose=1,
validation_data=(X_val, y_val, W_val), # test how the model predict unseen ratings
callbacks=[tensorboard_callback]
)
# Use TensorBoard to visualize metrics including loss and accuracy.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
history = model.fit(
x=X_train,
y=y_train,
sample_weight=W_train,
batch_size=64,
epochs=epochs,
verbose=1,
validation_data=(X_val, y_val, W_val), # test how the model predict unseen ratings
callbacks=[tensorboard_callback]
)
Epoch 1/120 352/352 [==============================] - 6s 12ms/step - loss: 2.2001 - val_loss: 2.3079 Epoch 2/120 352/352 [==============================] - 3s 7ms/step - loss: 1.3278 - val_loss: 0.8586 Epoch 3/120 352/352 [==============================] - 2s 7ms/step - loss: 0.8464 - val_loss: 0.6984 Epoch 4/120 352/352 [==============================] - 2s 7ms/step - loss: 0.4505 - val_loss: 0.4468 Epoch 5/120 352/352 [==============================] - 2s 7ms/step - loss: 0.3144 - val_loss: 0.3257 Epoch 6/120 352/352 [==============================] - 2s 7ms/step - loss: 0.2595 - val_loss: 0.2718 Epoch 7/120 352/352 [==============================] - 2s 7ms/step - loss: 0.2323 - val_loss: 0.2482 Epoch 8/120 352/352 [==============================] - 2s 7ms/step - loss: 0.2148 - val_loss: 0.2257 Epoch 9/120 352/352 [==============================] - 2s 7ms/step - loss: 0.2023 - val_loss: 0.2133 Epoch 10/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1914 - val_loss: 0.2006 Epoch 11/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1829 - val_loss: 0.1917 Epoch 12/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1765 - val_loss: 0.1842 Epoch 13/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1709 - val_loss: 0.1778 Epoch 14/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1660 - val_loss: 0.1719 Epoch 15/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1612 - val_loss: 0.1670 Epoch 16/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1566 - val_loss: 0.1620 Epoch 17/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1522 - val_loss: 0.1576 Epoch 18/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1479 - val_loss: 0.1532 Epoch 19/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1438 - val_loss: 0.1491 Epoch 20/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1398 - val_loss: 0.1452 Epoch 21/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1359 - val_loss: 0.1412 Epoch 22/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1321 - val_loss: 0.1377 Epoch 23/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1285 - val_loss: 0.1341 Epoch 24/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1250 - val_loss: 0.1308 Epoch 25/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1215 - val_loss: 0.1276 Epoch 26/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1183 - val_loss: 0.1244 Epoch 27/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1151 - val_loss: 0.1213 Epoch 28/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1122 - val_loss: 0.1186 Epoch 29/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1092 - val_loss: 0.1159 Epoch 30/120 352/352 [==============================] - 2s 6ms/step - loss: 0.1066 - val_loss: 0.1135 Epoch 31/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1040 - val_loss: 0.1111 Epoch 32/120 352/352 [==============================] - 2s 7ms/step - loss: 0.1017 - val_loss: 0.1091 Epoch 33/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0993 - val_loss: 0.1061 Epoch 34/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0970 - val_loss: 0.1042 Epoch 35/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0944 - val_loss: 0.1027 Epoch 36/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0928 - val_loss: 0.0997 Epoch 37/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0912 - val_loss: 0.0980 Epoch 38/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0885 - val_loss: 0.0968 Epoch 39/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0864 - val_loss: 0.0943 Epoch 40/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0850 - val_loss: 0.0925 Epoch 41/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0835 - val_loss: 0.0916 Epoch 42/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0816 - val_loss: 0.0892 Epoch 43/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0793 - val_loss: 0.0887 Epoch 44/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0784 - val_loss: 0.0861 Epoch 45/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0760 - val_loss: 0.0849 Epoch 46/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0750 - val_loss: 0.0836 Epoch 47/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0737 - val_loss: 0.0830 Epoch 48/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0724 - val_loss: 0.0822 Epoch 49/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0711 - val_loss: 0.0804 Epoch 50/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0695 - val_loss: 0.0818 Epoch 51/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0695 - val_loss: 0.0784 Epoch 52/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0677 - val_loss: 0.0771 Epoch 53/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0660 - val_loss: 0.0761 Epoch 54/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0650 - val_loss: 0.0759 Epoch 55/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0648 - val_loss: 0.0755 Epoch 56/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0640 - val_loss: 0.0740 Epoch 57/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0622 - val_loss: 0.0734 Epoch 58/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0618 - val_loss: 0.0727 Epoch 59/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0607 - val_loss: 0.0716 Epoch 60/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0599 - val_loss: 0.0712 Epoch 61/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0593 - val_loss: 0.0711 Epoch 62/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0586 - val_loss: 0.0703 Epoch 63/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0584 - val_loss: 0.0702 Epoch 64/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0572 - val_loss: 0.0690 Epoch 65/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0564 - val_loss: 0.0686 Epoch 66/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0558 - val_loss: 0.0688 Epoch 67/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0557 - val_loss: 0.0678 Epoch 68/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0546 - val_loss: 0.0680 Epoch 69/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0545 - val_loss: 0.0681 Epoch 70/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0540 - val_loss: 0.0669 Epoch 71/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0534 - val_loss: 0.0673 Epoch 72/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0527 - val_loss: 0.0669 Epoch 73/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0525 - val_loss: 0.0663 Epoch 74/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0517 - val_loss: 0.0661 Epoch 75/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0518 - val_loss: 0.0659 Epoch 76/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0512 - val_loss: 0.0661 Epoch 77/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0508 - val_loss: 0.0655 Epoch 78/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0502 - val_loss: 0.0655 Epoch 79/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0505 - val_loss: 0.0662 Epoch 80/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0499 - val_loss: 0.0660 Epoch 81/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0494 - val_loss: 0.0650 Epoch 82/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0488 - val_loss: 0.0649 Epoch 83/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0488 - val_loss: 0.0650 Epoch 84/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0488 - val_loss: 0.0651 Epoch 85/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0484 - val_loss: 0.0648 Epoch 86/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0480 - val_loss: 0.0654 Epoch 87/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0477 - val_loss: 0.0649 Epoch 88/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0474 - val_loss: 0.0649 Epoch 89/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0472 - val_loss: 0.0648 Epoch 90/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0474 - val_loss: 0.0649 Epoch 91/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0468 - val_loss: 0.0647 Epoch 92/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0465 - val_loss: 0.0651 Epoch 93/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0463 - val_loss: 0.0649 Epoch 94/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0464 - val_loss: 0.0662 Epoch 95/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0461 - val_loss: 0.0650 Epoch 96/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0462 - val_loss: 0.0654 Epoch 97/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0458 - val_loss: 0.0650 Epoch 98/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0454 - val_loss: 0.0651 Epoch 99/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0459 - val_loss: 0.0652 Epoch 100/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0453 - val_loss: 0.0651 Epoch 101/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0449 - val_loss: 0.0651 Epoch 102/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0449 - val_loss: 0.0654 Epoch 103/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0447 - val_loss: 0.0654 Epoch 104/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0448 - val_loss: 0.0657 Epoch 105/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0447 - val_loss: 0.0655 Epoch 106/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0445 - val_loss: 0.0658 Epoch 107/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0447 - val_loss: 0.0657 Epoch 108/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0441 - val_loss: 0.0656 Epoch 109/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0441 - val_loss: 0.0661 Epoch 110/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0441 - val_loss: 0.0659 Epoch 111/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0439 - val_loss: 0.0662 Epoch 112/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0440 - val_loss: 0.0665 Epoch 113/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0437 - val_loss: 0.0664 Epoch 114/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0436 - val_loss: 0.0663 Epoch 115/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0437 - val_loss: 0.0682 Epoch 116/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0437 - val_loss: 0.0667 Epoch 117/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0433 - val_loss: 0.0665 Epoch 118/120 352/352 [==============================] - 2s 7ms/step - loss: 0.0431 - val_loss: 0.0668 Epoch 119/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0434 - val_loss: 0.0668 Epoch 120/120 352/352 [==============================] - 2s 6ms/step - loss: 0.0433 - val_loss: 0.0671
In [ ]:
Copied!
model.summary()
model.summary()
Model: "cf_net"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) multiple 500
embedding_1 (Embedding) multiple 5
embedding_2 (Embedding) multiple 500000
embedding_3 (Embedding) multiple 5000
=================================================================
Total params: 505,505
Trainable params: 505,505
Non-trainable params: 0
_________________________________________________________________
In [ ]:
Copied!
plot_model(model, show_shapes=True, show_layer_names=True, to_file='cf-model.png')
plot_model(model, show_shapes=True, show_layer_names=True, to_file='cf-model.png')
Out[ ]:
In [ ]:
Copied!
%matplotlib inline
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "test"], loc="upper left")
plt.show()
%matplotlib inline
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "test"], loc="upper left")
plt.show()
In [ ]:
Copied!
%load_ext tensorboard
%tensorboard --logdir logs
%load_ext tensorboard
%tensorboard --logdir logs
Output hidden; open in https://colab.research.google.com to view.
But we really care about in model stacking is to predict the test set (T) ...
In [ ]:
Copied!
from utils_sys import highlight
import cf_models as cm
dR0, dT0, dX0 = R.shape, T.shape, X.shape
print(f"[info] shape(T): {T.shape} || shape(R): {R.shape} => shape(X): {X.shape}")
analyzer = cm.analyze_reconstruction(model, X, L, Pc, n_train, p_threshold=p_threshold, policy_threshold=policy_threshold)
highlight("(MSE) Reestimate the entire rating matrix (X) with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=False)
highlight("(MSE) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=True)
# Some post-hoc checks
assert (R.shape == dR0) and (T.shape == dT0) and (X.shape == dX0)
# Check potential side effects: (X, R, T) should retain their shapes
print(f"[info] shape(T): {T.shape} || shape(R): {R.shape} => shape(X): {X.shape}")
from utils_sys import highlight
import cf_models as cm
dR0, dT0, dX0 = R.shape, T.shape, X.shape
print(f"[info] shape(T): {T.shape} || shape(R): {R.shape} => shape(X): {X.shape}")
analyzer = cm.analyze_reconstruction(model, X, L, Pc, n_train, p_threshold=p_threshold, policy_threshold=policy_threshold)
highlight("(MSE) Reestimate the entire rating matrix (X) with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=False)
highlight("(MSE) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=True)
# Some post-hoc checks
assert (R.shape == dR0) and (T.shape == dT0) and (X.shape == dX0)
# Check potential side effects: (X, R, T) should retain their shapes
print(f"[info] shape(T): {T.shape} || shape(R): {R.shape} => shape(X): {X.shape}")
2.8.0 [info] shape(T): (5, 1250) || shape(R): (5, 3750) => shape(X): (5, 5000) ================================================================================ (MSE) Reestimate the entire rating matrix (X) with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 32.227823210244765 [info] From T to Th, delta(Frobenius norm)= 15.38273473416004 [info] How different are lh and lh_new? 0.1 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.1807909604519774 [result] Majority vote: F1 score with re-estimated Th: 0.17532467532467533 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.17877094972067037 [result] Best settings (complete): lh_maxvote, score: 0.18579234972677597 ================================================================================ (MSE) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 30.368335340354815 [info] From T to Th, delta(Frobenius norm)= 13.47236743885954 [info] How different are lh and lh_new? 0.012 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th: 0.1818181818181818 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.18378378378378382 [result] Best settings (unreliable only): lh_maxvote, score: 0.18579234972677597 [info] shape(T): (5, 1250) || shape(R): (5, 3750) => shape(X): (5, 5000)
In [ ]:
Copied!
# Check potential side effects: (X, R, T) should retain their shapes
print(f"[info] shape(T): {T.shape} || shape(R): {R.shape} => shape(X): {X.shape}")
# Check potential side effects: (X, R, T) should retain their shapes
print(f"[info] shape(T): {T.shape} || shape(R): {R.shape} => shape(X): {X.shape}")
[info] shape(T): (5, 1250) || shape(R): (5, 3750) => shape(X): (5, 5000)
Experiment #2: CFNet using BCE loss¶
- In this case, we use latent factors to approximate the labeling entailed by X (given the base predictor's probability thresholds)
2A: Preparing data for CFNet with BCE loss¶
In [ ]:
Copied!
uniq_colors = np.unique(Pc.A)
assert len(uniq_colors) == 4, f"n_colors: {uniq_colors}"
# Convert probability scores to label predicitons (0 and 1) in order to use BCE loss
# P = uc.to_preference(Pc)
C = Cn # use masked and rescaled confidence matrix
Lh = uc.estimateLabelMatrix(X, p_th=p_threshold)
Xc, yc, _, _ = dp.matrix_to_augmented_training_data(Lh, C, Pc) # NOTE: Don't overwrite X (`Xc` is not the same as `X`, which is a rating matrix)
sample_weights = dp.unravel(C, normalize=False) # Cn is a masked and balanced version of C0
assert len(sample_weights) == Xc.shape[0]
assert len(np.unique(yc)) == 2
##########################################
test_size = 0.1
# Make train-test split with sample weights
split_pt = int((1-test_size) * Xc.shape[0])
X_train, X_val, y_train, y_val, W_train, W_val = (
Xc[:split_pt],
Xc[split_pt:],
yc[:split_pt],
yc[split_pt:],
sample_weights[:split_pt],
sample_weights[split_pt:]
)
uniq_colors = np.unique(Pc.A)
assert len(uniq_colors) == 4, f"n_colors: {uniq_colors}"
# Convert probability scores to label predicitons (0 and 1) in order to use BCE loss
# P = uc.to_preference(Pc)
C = Cn # use masked and rescaled confidence matrix
Lh = uc.estimateLabelMatrix(X, p_th=p_threshold)
Xc, yc, _, _ = dp.matrix_to_augmented_training_data(Lh, C, Pc) # NOTE: Don't overwrite X (`Xc` is not the same as `X`, which is a rating matrix)
sample_weights = dp.unravel(C, normalize=False) # Cn is a masked and balanced version of C0
assert len(sample_weights) == Xc.shape[0]
assert len(np.unique(yc)) == 2
##########################################
test_size = 0.1
# Make train-test split with sample weights
split_pt = int((1-test_size) * Xc.shape[0])
X_train, X_val, y_train, y_val, W_train, W_val = (
Xc[:split_pt],
Xc[split_pt:],
yc[:split_pt],
yc[split_pt:],
sample_weights[:split_pt],
sample_weights[split_pt:]
)
2B. Create the model¶
- For convenience, all models and related utility funcitons are organized in the
cf_modelsmodule.
In [ ]:
Copied!
import cf_models as cm
n_factors = 100
epochs = 250 # BCE loss needs to train a bit longer
model = cm.get_cfnet_approximating_labels(n_users, n_items, n_factors)
import cf_models as cm
n_factors = 100
epochs = 250 # BCE loss needs to train a bit longer
model = cm.get_cfnet_approximating_labels(n_users, n_items, n_factors)
2C. Train and evaluate the model¶
In [ ]:
Copied!
# Use TensorBoard to visualize metrics including loss and accuracy.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
history = model.fit(
x=X_train,
y=y_train,
sample_weight=W_train,
batch_size=64,
epochs=epochs,
verbose=1,
validation_data=(X_val, y_val, W_val), # test how the model predict unseen ratings
callbacks=[tensorboard_callback]
)
# Use TensorBoard to visualize metrics including loss and accuracy.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
history = model.fit(
x=X_train,
y=y_train,
sample_weight=W_train,
batch_size=64,
epochs=epochs,
verbose=1,
validation_data=(X_val, y_val, W_val), # test how the model predict unseen ratings
callbacks=[tensorboard_callback]
)
Epoch 1/250 352/352 [==============================] - 4s 8ms/step - loss: 3.6048 - val_loss: 4.0405 Epoch 2/250 352/352 [==============================] - 2s 7ms/step - loss: 4.4791 - val_loss: 5.5703 Epoch 3/250 352/352 [==============================] - 2s 6ms/step - loss: 5.0250 - val_loss: 4.0120 Epoch 4/250 352/352 [==============================] - 2s 6ms/step - loss: 3.1053 - val_loss: 2.3680 Epoch 5/250 352/352 [==============================] - 2s 6ms/step - loss: 1.9987 - val_loss: 1.9047 Epoch 6/250 352/352 [==============================] - 2s 6ms/step - loss: 1.5588 - val_loss: 1.6602 Epoch 7/250 352/352 [==============================] - 2s 6ms/step - loss: 1.3731 - val_loss: 1.5153 Epoch 8/250 352/352 [==============================] - 2s 7ms/step - loss: 1.2681 - val_loss: 1.4249 Epoch 9/250 352/352 [==============================] - 2s 6ms/step - loss: 1.1949 - val_loss: 1.3594 Epoch 10/250 352/352 [==============================] - 2s 6ms/step - loss: 1.1482 - val_loss: 1.3231 Epoch 11/250 352/352 [==============================] - 2s 6ms/step - loss: 1.1109 - val_loss: 1.2841 Epoch 12/250 352/352 [==============================] - 2s 7ms/step - loss: 1.0793 - val_loss: 1.2513 Epoch 13/250 352/352 [==============================] - 2s 7ms/step - loss: 1.0435 - val_loss: 1.2166 Epoch 14/250 352/352 [==============================] - 2s 7ms/step - loss: 1.0190 - val_loss: 1.1865 Epoch 15/250 352/352 [==============================] - 2s 7ms/step - loss: 0.9903 - val_loss: 1.1507 Epoch 16/250 352/352 [==============================] - 2s 7ms/step - loss: 0.9619 - val_loss: 1.1231 Epoch 17/250 352/352 [==============================] - 2s 6ms/step - loss: 0.9324 - val_loss: 1.0914 Epoch 18/250 352/352 [==============================] - 2s 6ms/step - loss: 0.9047 - val_loss: 1.0633 Epoch 19/250 352/352 [==============================] - 2s 6ms/step - loss: 0.8793 - val_loss: 1.0376 Epoch 20/250 352/352 [==============================] - 2s 6ms/step - loss: 0.8558 - val_loss: 1.0131 Epoch 21/250 352/352 [==============================] - 2s 7ms/step - loss: 0.8332 - val_loss: 0.9900 Epoch 22/250 352/352 [==============================] - 2s 7ms/step - loss: 0.8118 - val_loss: 0.9675 Epoch 23/250 352/352 [==============================] - 2s 7ms/step - loss: 0.7915 - val_loss: 0.9465 Epoch 24/250 352/352 [==============================] - 2s 7ms/step - loss: 0.7720 - val_loss: 0.9255 Epoch 25/250 352/352 [==============================] - 2s 7ms/step - loss: 0.7529 - val_loss: 0.9049 Epoch 26/250 352/352 [==============================] - 2s 7ms/step - loss: 0.7341 - val_loss: 0.8847 Epoch 27/250 352/352 [==============================] - 2s 7ms/step - loss: 0.7159 - val_loss: 0.8650 Epoch 28/250 352/352 [==============================] - 2s 7ms/step - loss: 0.6988 - val_loss: 0.8464 Epoch 29/250 352/352 [==============================] - 2s 7ms/step - loss: 0.6825 - val_loss: 0.8297 Epoch 30/250 352/352 [==============================] - 2s 7ms/step - loss: 0.6663 - val_loss: 0.8083 Epoch 31/250 352/352 [==============================] - 2s 7ms/step - loss: 0.6482 - val_loss: 0.7926 Epoch 32/250 352/352 [==============================] - 2s 6ms/step - loss: 0.6371 - val_loss: 0.7736 Epoch 33/250 352/352 [==============================] - 2s 6ms/step - loss: 0.6164 - val_loss: 0.7541 Epoch 34/250 352/352 [==============================] - 2s 7ms/step - loss: 0.6030 - val_loss: 0.7426 Epoch 35/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5964 - val_loss: 0.7328 Epoch 36/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5756 - val_loss: 0.7049 Epoch 37/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5585 - val_loss: 0.6924 Epoch 38/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5445 - val_loss: 0.6765 Epoch 39/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5314 - val_loss: 0.6644 Epoch 40/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5191 - val_loss: 0.6390 Epoch 41/250 352/352 [==============================] - 2s 7ms/step - loss: 0.5005 - val_loss: 0.6285 Epoch 42/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4944 - val_loss: 0.6135 Epoch 43/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4778 - val_loss: 0.6007 Epoch 44/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4666 - val_loss: 0.5817 Epoch 45/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4496 - val_loss: 0.5671 Epoch 46/250 352/352 [==============================] - 2s 6ms/step - loss: 0.4423 - val_loss: 0.5553 Epoch 47/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4323 - val_loss: 0.5754 Epoch 48/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4219 - val_loss: 0.5298 Epoch 49/250 352/352 [==============================] - 2s 7ms/step - loss: 0.4072 - val_loss: 0.5153 Epoch 50/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3974 - val_loss: 0.5003 Epoch 51/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3824 - val_loss: 0.4880 Epoch 52/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3726 - val_loss: 0.4787 Epoch 53/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3640 - val_loss: 0.4660 Epoch 54/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3541 - val_loss: 0.4553 Epoch 55/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3462 - val_loss: 0.4461 Epoch 56/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3408 - val_loss: 0.4334 Epoch 57/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3272 - val_loss: 0.4210 Epoch 58/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3172 - val_loss: 0.4138 Epoch 59/250 352/352 [==============================] - 2s 7ms/step - loss: 0.3147 - val_loss: 0.4024 Epoch 60/250 352/352 [==============================] - 2s 6ms/step - loss: 0.3000 - val_loss: 0.3909 Epoch 61/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2922 - val_loss: 0.3826 Epoch 62/250 352/352 [==============================] - 2s 6ms/step - loss: 0.2844 - val_loss: 0.3723 Epoch 63/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2768 - val_loss: 0.3634 Epoch 64/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2698 - val_loss: 0.3541 Epoch 65/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2626 - val_loss: 0.3490 Epoch 66/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2564 - val_loss: 0.3425 Epoch 67/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2510 - val_loss: 0.3321 Epoch 68/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2416 - val_loss: 0.3207 Epoch 69/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2351 - val_loss: 0.3175 Epoch 70/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2291 - val_loss: 0.3068 Epoch 71/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2224 - val_loss: 0.2979 Epoch 72/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2167 - val_loss: 0.2907 Epoch 73/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2113 - val_loss: 0.2843 Epoch 74/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2057 - val_loss: 0.2785 Epoch 75/250 352/352 [==============================] - 2s 7ms/step - loss: 0.2003 - val_loss: 0.2720 Epoch 76/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1940 - val_loss: 0.2634 Epoch 77/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1887 - val_loss: 0.2572 Epoch 78/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1841 - val_loss: 0.2518 Epoch 79/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1792 - val_loss: 0.2451 Epoch 80/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1736 - val_loss: 0.2384 Epoch 81/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1690 - val_loss: 0.2334 Epoch 82/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1650 - val_loss: 0.2271 Epoch 83/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1599 - val_loss: 0.2218 Epoch 84/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1558 - val_loss: 0.2167 Epoch 85/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1518 - val_loss: 0.2115 Epoch 86/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1480 - val_loss: 0.2062 Epoch 87/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1432 - val_loss: 0.2013 Epoch 88/250 352/352 [==============================] - 3s 7ms/step - loss: 0.1397 - val_loss: 0.1965 Epoch 89/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1355 - val_loss: 0.1913 Epoch 90/250 352/352 [==============================] - 2s 6ms/step - loss: 0.1319 - val_loss: 0.1868 Epoch 91/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1285 - val_loss: 0.1828 Epoch 92/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1252 - val_loss: 0.1787 Epoch 93/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1216 - val_loss: 0.1737 Epoch 94/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1180 - val_loss: 0.1698 Epoch 95/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1150 - val_loss: 0.1658 Epoch 96/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1122 - val_loss: 0.1626 Epoch 97/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1089 - val_loss: 0.1579 Epoch 98/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1057 - val_loss: 0.1541 Epoch 99/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1030 - val_loss: 0.1505 Epoch 100/250 352/352 [==============================] - 2s 7ms/step - loss: 0.1002 - val_loss: 0.1471 Epoch 101/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0975 - val_loss: 0.1437 Epoch 102/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0948 - val_loss: 0.1400 Epoch 103/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0923 - val_loss: 0.1368 Epoch 104/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0897 - val_loss: 0.1335 Epoch 105/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0874 - val_loss: 0.1308 Epoch 106/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0852 - val_loss: 0.1276 Epoch 107/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0826 - val_loss: 0.1243 Epoch 108/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0804 - val_loss: 0.1214 Epoch 109/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0782 - val_loss: 0.1187 Epoch 110/250 352/352 [==============================] - 2s 6ms/step - loss: 0.0761 - val_loss: 0.1160 Epoch 111/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0741 - val_loss: 0.1134 Epoch 112/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0722 - val_loss: 0.1107 Epoch 113/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0701 - val_loss: 0.1083 Epoch 114/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0682 - val_loss: 0.1058 Epoch 115/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0664 - val_loss: 0.1034 Epoch 116/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0646 - val_loss: 0.1012 Epoch 117/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0630 - val_loss: 0.0989 Epoch 118/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0611 - val_loss: 0.0965 Epoch 119/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0595 - val_loss: 0.0944 Epoch 120/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0579 - val_loss: 0.0923 Epoch 121/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0564 - val_loss: 0.0903 Epoch 122/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0548 - val_loss: 0.0884 Epoch 123/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0534 - val_loss: 0.0863 Epoch 124/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0519 - val_loss: 0.0844 Epoch 125/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0505 - val_loss: 0.0826 Epoch 126/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0491 - val_loss: 0.0808 Epoch 127/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0479 - val_loss: 0.0789 Epoch 128/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0465 - val_loss: 0.0772 Epoch 129/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0453 - val_loss: 0.0756 Epoch 130/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0441 - val_loss: 0.0740 Epoch 131/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0429 - val_loss: 0.0724 Epoch 132/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0418 - val_loss: 0.0709 Epoch 133/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0406 - val_loss: 0.0694 Epoch 134/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0396 - val_loss: 0.0681 Epoch 135/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0385 - val_loss: 0.0666 Epoch 136/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0375 - val_loss: 0.0652 Epoch 137/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0365 - val_loss: 0.0638 Epoch 138/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0355 - val_loss: 0.0625 Epoch 139/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0346 - val_loss: 0.0612 Epoch 140/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0336 - val_loss: 0.0600 Epoch 141/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0327 - val_loss: 0.0588 Epoch 142/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0319 - val_loss: 0.0576 Epoch 143/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0310 - val_loss: 0.0564 Epoch 144/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0302 - val_loss: 0.0554 Epoch 145/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0294 - val_loss: 0.0542 Epoch 146/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0286 - val_loss: 0.0532 Epoch 147/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0279 - val_loss: 0.0521 Epoch 148/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0271 - val_loss: 0.0511 Epoch 149/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0264 - val_loss: 0.0500 Epoch 150/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0257 - val_loss: 0.0491 Epoch 151/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0250 - val_loss: 0.0482 Epoch 152/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0244 - val_loss: 0.0473 Epoch 153/250 352/352 [==============================] - 2s 6ms/step - loss: 0.0237 - val_loss: 0.0463 Epoch 154/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0231 - val_loss: 0.0455 Epoch 155/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0225 - val_loss: 0.0447 Epoch 156/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0219 - val_loss: 0.0439 Epoch 157/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0213 - val_loss: 0.0431 Epoch 158/250 352/352 [==============================] - 2s 6ms/step - loss: 0.0208 - val_loss: 0.0423 Epoch 159/250 352/352 [==============================] - 2s 6ms/step - loss: 0.0202 - val_loss: 0.0415 Epoch 160/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0197 - val_loss: 0.0407 Epoch 161/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0192 - val_loss: 0.0401 Epoch 162/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0187 - val_loss: 0.0393 Epoch 163/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0182 - val_loss: 0.0386 Epoch 164/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0177 - val_loss: 0.0380 Epoch 165/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0172 - val_loss: 0.0374 Epoch 166/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0168 - val_loss: 0.0367 Epoch 167/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0163 - val_loss: 0.0361 Epoch 168/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0159 - val_loss: 0.0355 Epoch 169/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0155 - val_loss: 0.0348 Epoch 170/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0151 - val_loss: 0.0343 Epoch 171/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0147 - val_loss: 0.0338 Epoch 172/250 352/352 [==============================] - 2s 6ms/step - loss: 0.0143 - val_loss: 0.0332 Epoch 173/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0139 - val_loss: 0.0327 Epoch 174/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0136 - val_loss: 0.0322 Epoch 175/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0132 - val_loss: 0.0317 Epoch 176/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0129 - val_loss: 0.0312 Epoch 177/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0126 - val_loss: 0.0307 Epoch 178/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0122 - val_loss: 0.0302 Epoch 179/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0119 - val_loss: 0.0297 Epoch 180/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0116 - val_loss: 0.0293 Epoch 181/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0113 - val_loss: 0.0288 Epoch 182/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0110 - val_loss: 0.0284 Epoch 183/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0107 - val_loss: 0.0280 Epoch 184/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0105 - val_loss: 0.0276 Epoch 185/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0102 - val_loss: 0.0272 Epoch 186/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0099 - val_loss: 0.0268 Epoch 187/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0097 - val_loss: 0.0265 Epoch 188/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0094 - val_loss: 0.0262 Epoch 189/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0092 - val_loss: 0.0257 Epoch 190/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0089 - val_loss: 0.0254 Epoch 191/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0087 - val_loss: 0.0250 Epoch 192/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0085 - val_loss: 0.0247 Epoch 193/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0083 - val_loss: 0.0244 Epoch 194/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0081 - val_loss: 0.0241 Epoch 195/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0079 - val_loss: 0.0238 Epoch 196/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0077 - val_loss: 0.0234 Epoch 197/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0075 - val_loss: 0.0232 Epoch 198/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0073 - val_loss: 0.0229 Epoch 199/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0071 - val_loss: 0.0226 Epoch 200/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0069 - val_loss: 0.0223 Epoch 201/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0067 - val_loss: 0.0221 Epoch 202/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0066 - val_loss: 0.0218 Epoch 203/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0064 - val_loss: 0.0216 Epoch 204/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0062 - val_loss: 0.0213 Epoch 205/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0061 - val_loss: 0.0211 Epoch 206/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0059 - val_loss: 0.0208 Epoch 207/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0058 - val_loss: 0.0206 Epoch 208/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0056 - val_loss: 0.0204 Epoch 209/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0055 - val_loss: 0.0202 Epoch 210/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0053 - val_loss: 0.0199 Epoch 211/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0052 - val_loss: 0.0197 Epoch 212/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0051 - val_loss: 0.0195 Epoch 213/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0050 - val_loss: 0.0193 Epoch 214/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0048 - val_loss: 0.0191 Epoch 215/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0047 - val_loss: 0.0189 Epoch 216/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0046 - val_loss: 0.0187 Epoch 217/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0045 - val_loss: 0.0185 Epoch 218/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0044 - val_loss: 0.0184 Epoch 219/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0043 - val_loss: 0.0182 Epoch 220/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0042 - val_loss: 0.0180 Epoch 221/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0040 - val_loss: 0.0179 Epoch 222/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0039 - val_loss: 0.0177 Epoch 223/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0038 - val_loss: 0.0175 Epoch 224/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0038 - val_loss: 0.0174 Epoch 225/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0037 - val_loss: 0.0172 Epoch 226/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0036 - val_loss: 0.0170 Epoch 227/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0035 - val_loss: 0.0170 Epoch 228/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0034 - val_loss: 0.0168 Epoch 229/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0033 - val_loss: 0.0167 Epoch 230/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0032 - val_loss: 0.0165 Epoch 231/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0032 - val_loss: 0.0164 Epoch 232/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0031 - val_loss: 0.0162 Epoch 233/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0030 - val_loss: 0.0161 Epoch 234/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0029 - val_loss: 0.0160 Epoch 235/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0029 - val_loss: 0.0159 Epoch 236/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0028 - val_loss: 0.0158 Epoch 237/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0027 - val_loss: 0.0156 Epoch 238/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0027 - val_loss: 0.0155 Epoch 239/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0026 - val_loss: 0.0154 Epoch 240/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0025 - val_loss: 0.0153 Epoch 241/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0025 - val_loss: 0.0152 Epoch 242/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0024 - val_loss: 0.0151 Epoch 243/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0023 - val_loss: 0.0150 Epoch 244/250 352/352 [==============================] - 3s 7ms/step - loss: 0.0023 - val_loss: 0.0149 Epoch 245/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0022 - val_loss: 0.0148 Epoch 246/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0022 - val_loss: 0.0147 Epoch 247/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0021 - val_loss: 0.0146 Epoch 248/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0021 - val_loss: 0.0145 Epoch 249/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0020 - val_loss: 0.0144 Epoch 250/250 352/352 [==============================] - 2s 7ms/step - loss: 0.0020 - val_loss: 0.0143
In [ ]:
Copied!
%matplotlib inline
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "test"], loc="upper left")
plt.show()
%matplotlib inline
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "test"], loc="upper left")
plt.show()
In [ ]:
Copied!
%load_ext tensorboard
%tensorboard --logdir logs
%load_ext tensorboard
%tensorboard --logdir logs
Output hidden; open in https://colab.research.google.com to view.
In [ ]:
Copied!
analyzer = cm.analyze_reconstruction(model, X, L, Pc, n_train, p_threshold=p_threshold, policy_threshold=policy_threshold)
highlight("(BCE) Reestimate the entire rating matrix (X) with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=False)
highlight("(BCE) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=True)
analyzer = cm.analyze_reconstruction(model, X, L, Pc, n_train, p_threshold=p_threshold, policy_threshold=policy_threshold)
highlight("(BCE) Reestimate the entire rating matrix (X) with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=False)
highlight("(BCE) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=True)
================================================================================ (BCE) Reestimate the entire rating matrix (X) with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 53.051820177848285 [info] From T to Th, delta(Frobenius norm)= 25.419679014738062 [info] How different are lh and lh_new? 0.0 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th: 0.18579234972677597 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.18579234972677597 [result] Best settings (complete): lh_maxvote, score: 0.18579234972677597 ================================================================================ (BCE) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 45.78218534396071 [info] From T to Th, delta(Frobenius norm)= 16.581544894914572 [info] How different are lh and lh_new? 0.0 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th: 0.18579234972677597 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.18579234972677597 [result] Best settings (unreliable only): lh_maxvote, score: 0.18579234972677597
Experiment #3: CFNet using confidence-weighted, color-specific loss (or "C-squared" loss)¶
- This loss function takes into account both the confidence weights and color matrix
- We will use the raw confidence matrix
C0instead of the balanced-and-rescaled confidence matrixCnbecause we also need to take into account FPs and FNs
3A. Preparing data for CFNet with the C-squared loss¶
In [ ]:
Copied!
# import cf_models as cm
assert len(np.unique(Pc.A)) == 4
# Use Cw as weights
Xc, yc, weights, colors = dp.matrix_to_augmented_training_data(X, Cw, Pc) # NOTE: Don't overwrite X (`Xc` is not the same as `X`, which is a rating matrix)
yc = np.column_stack([yc, weights, colors])
# We won't be using sample weights here because they are already being incorporated into the loss
# sample_weights = dp.unravel(Cn, normalize=False) # Cn is a masked and balanced version of C0
# C0 is a raw conficence matrix and should be dense
n_zeros = np.sum(weights == 0)
ratio_zeros = n_zeros/Cw.size
print(f"[info] How many zero weights in C0? {n_zeros}. The ratio of 0s should be small if not zero: {ratio_zeros}")
##########################################
test_size = 0.1
# Make train-test split with sample weights
split_pt = int((1-test_size) * Xc.shape[0])
X_train, X_val, y_train, y_val = (
Xc[:split_pt],
Xc[split_pt:],
yc[:split_pt],
yc[split_pt:],
)
# import cf_models as cm
assert len(np.unique(Pc.A)) == 4
# Use Cw as weights
Xc, yc, weights, colors = dp.matrix_to_augmented_training_data(X, Cw, Pc) # NOTE: Don't overwrite X (`Xc` is not the same as `X`, which is a rating matrix)
yc = np.column_stack([yc, weights, colors])
# We won't be using sample weights here because they are already being incorporated into the loss
# sample_weights = dp.unravel(Cn, normalize=False) # Cn is a masked and balanced version of C0
# C0 is a raw conficence matrix and should be dense
n_zeros = np.sum(weights == 0)
ratio_zeros = n_zeros/Cw.size
print(f"[info] How many zero weights in C0? {n_zeros}. The ratio of 0s should be small if not zero: {ratio_zeros}")
##########################################
test_size = 0.1
# Make train-test split with sample weights
split_pt = int((1-test_size) * Xc.shape[0])
X_train, X_val, y_train, y_val = (
Xc[:split_pt],
Xc[split_pt:],
yc[:split_pt],
yc[split_pt:],
)
[info] How many zero weights in C0? 505. The ratio of 0s should be small if not zero: 0.020616452337211676
3B. Create the model¶
In [ ]:
Copied!
n_factors = 100
epochs = 100
n_users, n_items = X.shape
loss_fn = cm.confidence_weighted_loss
model = cm.get_cfnet_uncompiled(n_users, n_items, n_factors)
model.compile(
loss=loss_fn, optimizer=keras.optimizers.Adam(lr=0.001)
)
n_factors = 100
epochs = 100
n_users, n_items = X.shape
loss_fn = cm.confidence_weighted_loss
model = cm.get_cfnet_uncompiled(n_users, n_items, n_factors)
model.compile(
loss=loss_fn, optimizer=keras.optimizers.Adam(lr=0.001)
)
3C. Train and evaluate the model¶
In [ ]:
Copied!
# Use TensorBoard to visualize metrics including loss and accuracy.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
history = model.fit(
x=X_train,
y=y_train,
# sample_weight=W_train, # not using sample weight in this case
batch_size=64,
epochs=epochs,
verbose=1,
validation_data=(X_val, y_val), # test how the model predict unseen ratings
callbacks=[tensorboard_callback]
)
# Use TensorBoard to visualize metrics including loss and accuracy.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
history = model.fit(
x=X_train,
y=y_train,
# sample_weight=W_train, # not using sample weight in this case
batch_size=64,
epochs=epochs,
verbose=1,
validation_data=(X_val, y_val), # test how the model predict unseen ratings
callbacks=[tensorboard_callback]
)
Epoch 1/100 352/352 [==============================] - 4s 7ms/step - loss: 1.9887 - val_loss: 1.7814 Epoch 2/100 352/352 [==============================] - 2s 6ms/step - loss: 1.1355 - val_loss: 0.9377 Epoch 3/100 352/352 [==============================] - 2s 7ms/step - loss: 0.7805 - val_loss: 0.5668 Epoch 4/100 352/352 [==============================] - 2s 6ms/step - loss: 0.4348 - val_loss: 0.4167 Epoch 5/100 352/352 [==============================] - 2s 6ms/step - loss: 0.3375 - val_loss: 0.3238 Epoch 6/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2935 - val_loss: 0.2975 Epoch 7/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2755 - val_loss: 0.2773 Epoch 8/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2653 - val_loss: 0.2636 Epoch 9/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2581 - val_loss: 0.2548 Epoch 10/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2545 - val_loss: 0.2500 Epoch 11/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2533 - val_loss: 0.2465 Epoch 12/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2531 - val_loss: 0.2448 Epoch 13/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2530 - val_loss: 0.2443 Epoch 14/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2529 - val_loss: 0.2445 Epoch 15/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2529 - val_loss: 0.2444 Epoch 16/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2529 - val_loss: 0.2444 Epoch 17/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2529 - val_loss: 0.2441 Epoch 18/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2529 - val_loss: 0.2437 Epoch 19/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2529 - val_loss: 0.2438 Epoch 20/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2529 - val_loss: 0.2438 Epoch 21/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2528 - val_loss: 0.2440 Epoch 22/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2533 - val_loss: 0.2444 Epoch 23/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2530 - val_loss: 0.2440 Epoch 24/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2531 - val_loss: 0.2436 Epoch 25/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2531 - val_loss: 0.2434 Epoch 26/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2542 - val_loss: 0.2448 Epoch 27/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2539 - val_loss: 0.2446 Epoch 28/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2555 - val_loss: 0.2452 Epoch 29/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2562 - val_loss: 0.2440 Epoch 30/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2553 - val_loss: 0.2455 Epoch 31/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2563 - val_loss: 0.2463 Epoch 32/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2579 - val_loss: 0.2465 Epoch 33/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2582 - val_loss: 0.2532 Epoch 34/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2566 - val_loss: 0.2465 Epoch 35/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2548 - val_loss: 0.2452 Epoch 36/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2543 - val_loss: 0.2456 Epoch 37/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2555 - val_loss: 0.2545 Epoch 38/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2671 - val_loss: 0.2521 Epoch 39/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2586 - val_loss: 0.2442 Epoch 40/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2533 - val_loss: 0.2436 Epoch 41/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2536 - val_loss: 0.2441 Epoch 42/100 352/352 [==============================] - 3s 7ms/step - loss: 0.2574 - val_loss: 0.2607 Epoch 43/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2619 - val_loss: 0.2544 Epoch 44/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2578 - val_loss: 0.2433 Epoch 45/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2537 - val_loss: 0.2478 Epoch 46/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2555 - val_loss: 0.2455 Epoch 47/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2542 - val_loss: 0.2442 Epoch 48/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2575 - val_loss: 0.2524 Epoch 49/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2590 - val_loss: 0.2470 Epoch 50/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2571 - val_loss: 0.2440 Epoch 51/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2540 - val_loss: 0.2470 Epoch 52/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2592 - val_loss: 0.2506 Epoch 53/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2603 - val_loss: 0.2466 Epoch 54/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2558 - val_loss: 0.2448 Epoch 55/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2541 - val_loss: 0.2458 Epoch 56/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2546 - val_loss: 0.2452 Epoch 57/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2640 - val_loss: 0.2474 Epoch 58/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2545 - val_loss: 0.2446 Epoch 59/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2546 - val_loss: 0.2445 Epoch 60/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2608 - val_loss: 0.2456 Epoch 61/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2560 - val_loss: 0.2442 Epoch 62/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2544 - val_loss: 0.2434 Epoch 63/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2565 - val_loss: 0.2440 Epoch 64/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2628 - val_loss: 0.2520 Epoch 65/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2550 - val_loss: 0.2432 Epoch 66/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2539 - val_loss: 0.2479 Epoch 67/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2581 - val_loss: 0.2465 Epoch 68/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2581 - val_loss: 0.2440 Epoch 69/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2555 - val_loss: 0.2440 Epoch 70/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2575 - val_loss: 0.2526 Epoch 71/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2596 - val_loss: 0.2504 Epoch 72/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2558 - val_loss: 0.2436 Epoch 73/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2542 - val_loss: 0.2476 Epoch 74/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2574 - val_loss: 0.2601 Epoch 75/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2586 - val_loss: 0.2471 Epoch 76/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2581 - val_loss: 0.2441 Epoch 77/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2537 - val_loss: 0.2448 Epoch 78/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2564 - val_loss: 0.2534 Epoch 79/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2615 - val_loss: 0.2462 Epoch 80/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2541 - val_loss: 0.2445 Epoch 81/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2557 - val_loss: 0.2507 Epoch 82/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2583 - val_loss: 0.2456 Epoch 83/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2606 - val_loss: 0.2664 Epoch 84/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2571 - val_loss: 0.2441 Epoch 85/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2535 - val_loss: 0.2445 Epoch 86/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2553 - val_loss: 0.2448 Epoch 87/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2559 - val_loss: 0.2468 Epoch 88/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2593 - val_loss: 0.2625 Epoch 89/100 352/352 [==============================] - 3s 7ms/step - loss: 0.2607 - val_loss: 0.2441 Epoch 90/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2534 - val_loss: 0.2432 Epoch 91/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2537 - val_loss: 0.2438 Epoch 92/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2555 - val_loss: 0.2561 Epoch 93/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2637 - val_loss: 0.2520 Epoch 94/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2592 - val_loss: 0.2504 Epoch 95/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2576 - val_loss: 0.2460 Epoch 96/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2553 - val_loss: 0.2436 Epoch 97/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2533 - val_loss: 0.2450 Epoch 98/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2559 - val_loss: 0.2470 Epoch 99/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2608 - val_loss: 0.2534 Epoch 100/100 352/352 [==============================] - 3s 7ms/step - loss: 0.2576 - val_loss: 0.2448
In [ ]:
Copied!
%matplotlib inline
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "test"], loc="upper left")
plt.show()
%matplotlib inline
f, ax1 = plt.subplots(nrows=1, ncols=1,figsize=(20,8))
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "test"], loc="upper left")
plt.show()
In [ ]:
Copied!
analyzer = cm.analyze_reconstruction(model, X, L, Pc, n_train, p_threshold=p_threshold, policy_threshold=policy_threshold)
highlight("(C-Sqr) Reestimate the entire rating matrix (X) with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=False)
highlight("(C-Sqr) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=True)
analyzer = cm.analyze_reconstruction(model, X, L, Pc, n_train, p_threshold=p_threshold, policy_threshold=policy_threshold)
highlight("(C-Sqr) Reestimate the entire rating matrix (X) with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=False)
highlight("(C-Sqr) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings")
reestimated = analyzer(L_test, unreliable_only=True)
================================================================================ (C-Sqr) Reestimate the entire rating matrix (X) with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 38.816793098497804 [info] From T to Th, delta(Frobenius norm)= 21.403534541221376 [info] How different are lh and lh_new? 0.2088 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.19364161849710984 [result] Majority vote: F1 score with re-estimated Th: 0.19457013574660634 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.18934911242603553 [result] Best settings (complete): lh2_maxvote_pth_adjusted, score: 0.19457013574660634 ================================================================================ (C-Sqr) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 22.8079703597208 [info] From T to Th, delta(Frobenius norm)= 10.000121918376424 [info] How different are lh and lh_new? 0.0072 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th: 0.18750000000000003 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.1925133689839572 [result] Best settings (unreliable only): lh2_maxvote_pth_adjusted, score: 0.18750000000000003
Experiment #4: CFNet using C-squared loss and sparse confidence matrix¶
- Use the sparse confidence matrix
Cninstead of the dense matrixCwin the optimization objective- Recall that
Cnholds confidence scores for TPs and TNs but those associated with FPs and FNs are masked (zeroed out so that they don't enter the optimization objective)
- Recall that
In [ ]:
Copied!
import cf_models as cm
from cf_models import confidence_weighted_loss
n_factors = 100
alpha = 100.0
conf_measure = 'brier'
policy_threshold = 'balanced' # 'fmax'
print(f"[info] input_dir:\n{input_dir}\n")
cm.demo_cfnet_with_csqr_loss(loss_fn=confidence_weighted_loss,
ctype='Cn',
n_factors=n_factors,
alpha=alpha,
conf_measure=conf_measure,
policy_threshold=policy_threshold, data_dir=os.path.join(input_dir, 'data'))
import cf_models as cm
from cf_models import confidence_weighted_loss
n_factors = 100
alpha = 100.0
conf_measure = 'brier'
policy_threshold = 'balanced' # 'fmax'
print(f"[info] input_dir:\n{input_dir}\n")
cm.demo_cfnet_with_csqr_loss(loss_fn=confidence_weighted_loss,
ctype='Cn',
n_factors=n_factors,
alpha=alpha,
conf_measure=conf_measure,
policy_threshold=policy_threshold, data_dir=os.path.join(input_dir, 'data'))
[info] input_dir: /content/drive/MyDrive/Colab Notebooks/machine_learning_examples/cf_ensemble [info] list of base classifiers: ['RF' 'KNNC' 'GNB' 'QDA' 'MLPClassifier'] (estimateProbThresholds) policy: fmax [info] probability thresholds: [0.499 0. 0.008 0. 0.007] > shape(R):(5, 3750) || shape(T): (5, 1250) => shape(X): (5, 5000) (make_cn) Using WEIGHTED confidence matrix to approximate ratings ... Epoch 1/100 352/352 [==============================] - 4s 7ms/step - loss: 2.1333 - val_loss: 1.6803 Epoch 2/100 352/352 [==============================] - 2s 6ms/step - loss: 1.1397 - val_loss: 0.8601 Epoch 3/100 352/352 [==============================] - 2s 6ms/step - loss: 0.7135 - val_loss: 0.4853 Epoch 4/100 352/352 [==============================] - 2s 7ms/step - loss: 0.3996 - val_loss: 0.3250 Epoch 5/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2881 - val_loss: 0.2488 Epoch 6/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2456 - val_loss: 0.2113 Epoch 7/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2244 - val_loss: 0.1942 Epoch 8/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2149 - val_loss: 0.1838 Epoch 9/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2100 - val_loss: 0.1771 Epoch 10/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2084 - val_loss: 0.1731 Epoch 11/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2078 - val_loss: 0.1711 Epoch 12/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2076 - val_loss: 0.1699 Epoch 13/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2076 - val_loss: 0.1683 Epoch 14/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2075 - val_loss: 0.1681 Epoch 15/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2075 - val_loss: 0.1669 Epoch 16/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2075 - val_loss: 0.1663 Epoch 17/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2075 - val_loss: 0.1664 Epoch 18/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2075 - val_loss: 0.1661 Epoch 19/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2074 - val_loss: 0.1664 Epoch 20/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2074 - val_loss: 0.1661 Epoch 21/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2076 - val_loss: 0.1657 Epoch 22/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2074 - val_loss: 0.1655 Epoch 23/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2073 - val_loss: 0.1663 Epoch 24/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2076 - val_loss: 0.1663 Epoch 25/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2077 - val_loss: 0.1666 Epoch 26/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2078 - val_loss: 0.1670 Epoch 27/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2080 - val_loss: 0.1678 Epoch 28/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2090 - val_loss: 0.1685 Epoch 29/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2084 - val_loss: 0.1678 Epoch 30/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2095 - val_loss: 0.1710 Epoch 31/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2099 - val_loss: 0.1705 Epoch 32/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2119 - val_loss: 0.1718 Epoch 33/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2090 - val_loss: 0.1740 Epoch 34/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2116 - val_loss: 0.1696 Epoch 35/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2097 - val_loss: 0.1675 Epoch 36/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2101 - val_loss: 0.1718 Epoch 37/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2132 - val_loss: 0.1699 Epoch 38/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2128 - val_loss: 0.1659 Epoch 39/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2078 - val_loss: 0.1661 Epoch 40/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2088 - val_loss: 0.1738 Epoch 41/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2100 - val_loss: 0.1696 Epoch 42/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2120 - val_loss: 0.1711 Epoch 43/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2125 - val_loss: 0.1675 Epoch 44/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2097 - val_loss: 0.1678 Epoch 45/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2086 - val_loss: 0.1660 Epoch 46/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2093 - val_loss: 0.1672 Epoch 47/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2094 - val_loss: 0.1674 Epoch 48/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2167 - val_loss: 0.1660 Epoch 49/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2084 - val_loss: 0.1667 Epoch 50/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2106 - val_loss: 0.1681 Epoch 51/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2106 - val_loss: 0.1678 Epoch 52/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2106 - val_loss: 0.1686 Epoch 53/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2108 - val_loss: 0.1695 Epoch 54/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2126 - val_loss: 0.1663 Epoch 55/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2079 - val_loss: 0.1673 Epoch 56/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2104 - val_loss: 0.1697 Epoch 57/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2086 - val_loss: 0.1678 Epoch 58/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2144 - val_loss: 0.1738 Epoch 59/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2141 - val_loss: 0.1674 Epoch 60/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2093 - val_loss: 0.1661 Epoch 61/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2077 - val_loss: 0.1670 Epoch 62/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2093 - val_loss: 0.1675 Epoch 63/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2119 - val_loss: 0.1693 Epoch 64/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2117 - val_loss: 0.1716 Epoch 65/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2110 - val_loss: 0.1684 Epoch 66/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2115 - val_loss: 0.1766 Epoch 67/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2087 - val_loss: 0.1677 Epoch 68/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2095 - val_loss: 0.1671 Epoch 69/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2119 - val_loss: 0.1678 Epoch 70/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2110 - val_loss: 0.1670 Epoch 71/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2097 - val_loss: 0.1668 Epoch 72/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2090 - val_loss: 0.1682 Epoch 73/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2139 - val_loss: 0.1689 Epoch 74/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2128 - val_loss: 0.1660 Epoch 75/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2078 - val_loss: 0.1660 Epoch 76/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2086 - val_loss: 0.1679 Epoch 77/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2118 - val_loss: 0.1676 Epoch 78/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2104 - val_loss: 0.1669 Epoch 79/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2093 - val_loss: 0.1747 Epoch 80/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2120 - val_loss: 0.1719 Epoch 81/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2155 - val_loss: 0.1681 Epoch 82/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2083 - val_loss: 0.1662 Epoch 83/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2077 - val_loss: 0.1661 Epoch 84/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2098 - val_loss: 0.1694 Epoch 85/100 352/352 [==============================] - 3s 9ms/step - loss: 0.2126 - val_loss: 0.1770 Epoch 86/100 352/352 [==============================] - 4s 10ms/step - loss: 0.2097 - val_loss: 0.1659 Epoch 87/100 352/352 [==============================] - 4s 11ms/step - loss: 0.2082 - val_loss: 0.1672 Epoch 88/100 352/352 [==============================] - 4s 11ms/step - loss: 0.2106 - val_loss: 0.1713 Epoch 89/100 352/352 [==============================] - 3s 9ms/step - loss: 0.2103 - val_loss: 0.1680 Epoch 90/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2087 - val_loss: 0.1663 Epoch 91/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2112 - val_loss: 0.1706 Epoch 92/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2130 - val_loss: 0.1699 Epoch 93/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2108 - val_loss: 0.1664 Epoch 94/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2089 - val_loss: 0.1681 Epoch 95/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2102 - val_loss: 0.1694 Epoch 96/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2115 - val_loss: 0.1699 Epoch 97/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2112 - val_loss: 0.1660 Epoch 98/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2099 - val_loss: 0.1676 Epoch 99/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2119 - val_loss: 0.1679 Epoch 100/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2117 - val_loss: 0.1666
================================================================================ (C-Sqr with Cn) Reestimate the entire rating matrix (X) with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 39.16099544127994 [info] From T to Th, delta(Frobenius norm)= 21.753993446623824 [info] How different are lh and lh_new? 0.2336 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.19364161849710984 [result] Majority vote: F1 score with re-estimated Th: 0.16414686825053992 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.17610062893081763 [result] Best settings (complete): lh2_maxvote_pth_unadjusted, score: 0.19364161849710984 ================================================================================ (C-Sqr with Cn) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 22.645645639657772 [info] From T to Th, delta(Frobenius norm)= 9.898313781800729 [info] How different are lh and lh_new? 0.0072 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th: 0.18750000000000003 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.1925133689839572 [result] Best settings (unreliable only): lh2_maxvote_pth_adjusted, score: 0.18750000000000003
Out[ ]:
<cf_models.CFNet at 0x7f334c0bf9d0>
Experiment #5: CFNet with another version of C-square loss¶
- Similar to experiment #3 but also use the probability thresholds to account for errors associated with FPs and FNs
- Unlike experiment #4, we'll use
Cwbecause we also need to account for FPs and FNs
In [ ]:
Copied!
import cf_models as cm
from cf_models import c_squared_loss
n_factors = 100
alpha = 100.0
conf_measure = 'brier'
policy_threshold = 'balanced' # 'fmax'
cm.demo_cfnet_with_csqr_loss(loss_fn=c_squared_loss,
ctype='Cw',
n_factors=n_factors,
alpha=alpha,
conf_measure=conf_measure,
policy_threshold=policy_threshold, data_dir=os.path.join(input_dir, 'data'))
import cf_models as cm
from cf_models import c_squared_loss
n_factors = 100
alpha = 100.0
conf_measure = 'brier'
policy_threshold = 'balanced' # 'fmax'
cm.demo_cfnet_with_csqr_loss(loss_fn=c_squared_loss,
ctype='Cw',
n_factors=n_factors,
alpha=alpha,
conf_measure=conf_measure,
policy_threshold=policy_threshold, data_dir=os.path.join(input_dir, 'data'))
[info] list of base classifiers: ['RF' 'KNNC' 'GNB' 'QDA' 'MLPClassifier'] (estimateProbThresholds) policy: fmax [info] probability thresholds: [0.499 0. 0.008 0. 0.007] > shape(R):(5, 3750) || shape(T): (5, 1250) => shape(X): (5, 5000) (make_cn) Using WEIGHTED confidence matrix to approximate ratings ... Epoch 1/100 352/352 [==============================] - 4s 7ms/step - loss: 2.2751 - val_loss: 1.6860 Epoch 2/100 352/352 [==============================] - 2s 7ms/step - loss: 1.2882 - val_loss: 0.9987 Epoch 3/100 352/352 [==============================] - 3s 7ms/step - loss: 0.8296 - val_loss: 0.5659 Epoch 4/100 352/352 [==============================] - 2s 7ms/step - loss: 0.4723 - val_loss: 0.3606 Epoch 5/100 352/352 [==============================] - 2s 7ms/step - loss: 0.3419 - val_loss: 0.2538 Epoch 6/100 352/352 [==============================] - 3s 7ms/step - loss: 0.2873 - val_loss: 0.2152 Epoch 7/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2674 - val_loss: 0.1936 Epoch 8/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2528 - val_loss: 0.1811 Epoch 9/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2444 - val_loss: 0.1708 Epoch 10/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2392 - val_loss: 0.1644 Epoch 11/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2374 - val_loss: 0.1612 Epoch 12/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2368 - val_loss: 0.1603 Epoch 13/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2368 - val_loss: 0.1596 Epoch 14/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1596 Epoch 15/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1591 Epoch 16/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1590 Epoch 17/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1594 Epoch 18/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2367 - val_loss: 0.1591 Epoch 19/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1588 Epoch 20/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2366 - val_loss: 0.1592 Epoch 21/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2366 - val_loss: 0.1589 Epoch 22/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1584 Epoch 23/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1583 Epoch 24/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2367 - val_loss: 0.1588 Epoch 25/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2369 - val_loss: 0.1589 Epoch 26/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2370 - val_loss: 0.1597 Epoch 27/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2369 - val_loss: 0.1591 Epoch 28/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2373 - val_loss: 0.1604 Epoch 29/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2374 - val_loss: 0.1593 Epoch 30/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2386 - val_loss: 0.1633 Epoch 31/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2386 - val_loss: 0.1594 Epoch 32/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2382 - val_loss: 0.1600 Epoch 33/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2405 - val_loss: 0.1633 Epoch 34/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2417 - val_loss: 0.1594 Epoch 35/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2380 - val_loss: 0.1606 Epoch 36/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2384 - val_loss: 0.1594 Epoch 37/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2393 - val_loss: 0.1613 Epoch 38/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2421 - val_loss: 0.1647 Epoch 39/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2386 - val_loss: 0.1626 Epoch 40/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2389 - val_loss: 0.1589 Epoch 41/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2404 - val_loss: 0.1616 Epoch 42/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2413 - val_loss: 0.1586 Epoch 43/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2384 - val_loss: 0.1582 Epoch 44/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2376 - val_loss: 0.1594 Epoch 45/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2420 - val_loss: 0.1657 Epoch 46/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2401 - val_loss: 0.1599 Epoch 47/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2376 - val_loss: 0.1593 Epoch 48/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2415 - val_loss: 0.1615 Epoch 49/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2402 - val_loss: 0.1705 Epoch 50/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2387 - val_loss: 0.1582 Epoch 51/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2377 - val_loss: 0.1580 Epoch 52/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2396 - val_loss: 0.1674 Epoch 53/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2441 - val_loss: 0.1601 Epoch 54/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2376 - val_loss: 0.1609 Epoch 55/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2405 - val_loss: 0.1620 Epoch 56/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2425 - val_loss: 0.1596 Epoch 57/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2391 - val_loss: 0.1589 Epoch 58/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2404 - val_loss: 0.1614 Epoch 59/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2387 - val_loss: 0.1624 Epoch 60/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2392 - val_loss: 0.1635 Epoch 61/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2407 - val_loss: 0.1592 Epoch 62/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2393 - val_loss: 0.1586 Epoch 63/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2399 - val_loss: 0.1656 Epoch 64/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2427 - val_loss: 0.1591 Epoch 65/100 352/352 [==============================] - 2s 6ms/step - loss: 0.2388 - val_loss: 0.1593 Epoch 66/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2383 - val_loss: 0.1591 Epoch 67/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2401 - val_loss: 0.1676 Epoch 68/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2412 - val_loss: 0.1607 Epoch 69/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2384 - val_loss: 0.1634 Epoch 70/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2407 - val_loss: 0.1587 Epoch 71/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2398 - val_loss: 0.1698 Epoch 72/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2400 - val_loss: 0.1613 Epoch 73/100 352/352 [==============================] - 3s 7ms/step - loss: 0.2410 - val_loss: 0.1636 Epoch 74/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2423 - val_loss: 0.1709 Epoch 75/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2392 - val_loss: 0.1609 Epoch 76/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2385 - val_loss: 0.1618 Epoch 77/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2387 - val_loss: 0.1632 Epoch 78/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2392 - val_loss: 0.1614 Epoch 79/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2392 - val_loss: 0.1595 Epoch 80/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2391 - val_loss: 0.1698 Epoch 81/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2434 - val_loss: 0.1586 Epoch 82/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2379 - val_loss: 0.1589 Epoch 83/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2382 - val_loss: 0.1605 Epoch 84/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2420 - val_loss: 0.1643 Epoch 85/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2429 - val_loss: 0.1623 Epoch 86/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2387 - val_loss: 0.1582 Epoch 87/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2371 - val_loss: 0.1581 Epoch 88/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2383 - val_loss: 0.1592 Epoch 89/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2414 - val_loss: 0.1623 Epoch 90/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2399 - val_loss: 0.1612 Epoch 91/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2397 - val_loss: 0.1609 Epoch 92/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2390 - val_loss: 0.1603 Epoch 93/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2423 - val_loss: 0.1598 Epoch 94/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2376 - val_loss: 0.1607 Epoch 95/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2395 - val_loss: 0.1598 Epoch 96/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2429 - val_loss: 0.1612 Epoch 97/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2375 - val_loss: 0.1592 Epoch 98/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2382 - val_loss: 0.1607 Epoch 99/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2397 - val_loss: 0.1598 Epoch 100/100 352/352 [==============================] - 2s 7ms/step - loss: 0.2407 - val_loss: 0.1775
================================================================================ (C-Sqr with Cw) Reestimate the entire rating matrix (X) with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 37.82624812811406 [info] From T to Th, delta(Frobenius norm)= 20.908288958328175 [info] How different are lh and lh_new? 0.192 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.19392185238784368 [result] Majority vote: F1 score with re-estimated Th: 0.20537897310513448 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.18181818181818182 [result] Best settings (complete): lh2_maxvote_pth_adjusted, score: 0.20537897310513448 ================================================================================ (C-Sqr with Cw) Reestimate ONLY the unreliable entries in X with learned latent factors/embeddings ================================================================================ [info] From R to Rh, delta(Frobenius norm)= 23.580730236422873 [info] From T to Th, delta(Frobenius norm)= 10.321363800157425 [info] How different are lh and lh_new? 0.0072 [result] Majority vote: F1 score with the original T: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th using original p_threshold: 0.18579234972677597 [result] Majority vote: F1 score with re-estimated Th: 0.18750000000000003 [result] Stacking: F1 score with the original T: 0.09859154929577464 [result] Stacking: F1 score with re-estimated Th: 0.1925133689839572 [result] Best settings (unreliable only): lh2_maxvote_pth_adjusted, score: 0.18750000000000003
Out[ ]:
<cf_models.CFNet at 0x7f334bbaf410>
Useful utility functions¶
- data_pipeline.iterporate()
In [ ]:
Copied!
# import data_pipeline as dp
# Interporate (NOT in the sense of its use in regression):
# useful for matrix reconstruction using latent factors
###############################
X = np.random.choice(range(1, 6), (3, 10))
T = np.repeat(99, 3*10).reshape(3, 10)
M = np.where(X%2==0, 1, 0)
print(f"X:\n{X}\n")
print(f"T:\n{T}\n")
print(f"M:\n{M}\n")
Xp = cm.interpolate(X, T, Pc=M)
print(f"Xp:\n{Xp}\n")
# Training data preparation
################################
D = np.random.choice(range(10), (5, 5))
print(D); print()
scores = dp.unravel(D, verify=1)
assert len(scores) == D.size
print(scores)
# import data_pipeline as dp
# Interporate (NOT in the sense of its use in regression):
# useful for matrix reconstruction using latent factors
###############################
X = np.random.choice(range(1, 6), (3, 10))
T = np.repeat(99, 3*10).reshape(3, 10)
M = np.where(X%2==0, 1, 0)
print(f"X:\n{X}\n")
print(f"T:\n{T}\n")
print(f"M:\n{M}\n")
Xp = cm.interpolate(X, T, Pc=M)
print(f"Xp:\n{Xp}\n")
# Training data preparation
################################
D = np.random.choice(range(10), (5, 5))
print(D); print()
scores = dp.unravel(D, verify=1)
assert len(scores) == D.size
print(scores)
X: [[1 1 4 2 1 4 4 5 1 2] [5 4 3 3 5 3 5 4 4 1] [4 3 2 2 3 3 5 2 4 5]] T: [[99 99 99 99 99 99 99 99 99 99] [99 99 99 99 99 99 99 99 99 99] [99 99 99 99 99 99 99 99 99 99]] M: [[0 0 1 1 0 1 1 0 0 1] [0 1 0 0 0 0 0 1 1 0] [1 0 1 1 0 0 0 1 1 0]] Xp: [[99. 99. 4. 2. 99. 4. 4. 99. 99. 2.] [99. 4. 99. 99. 99. 99. 99. 4. 4. 99.] [ 4. 99. 2. 2. 99. 99. 99. 2. 4. 99.]] [[9 3 0 1 3] [3 5 0 5 6] [0 5 9 7 4] [0 3 9 4 7] [0 8 2 6 9]] [9 3 0 1 3 3 5 0 5 6 0 5 9 7 4 0 3 9 4 7 0 8 2 6 9]
- From user-item pairs (representing implicit feedbacks), construct corresponding a sparse (rating) matrices
In [ ]:
Copied!
from scipy import sparse
# Simulate users rating on items
df_ui = pd.DataFrame([['a', 'abc'],['b', 'def'],['c', 'ghi'],
['d', 'abc'],['a', 'ghi'],['e', 'fg'],
['f', 'f76'],['b', 'f76']],
columns = ['user','item'])
display(df_ui)
UI = df_ui.values
print()
# I: row coordiante, J: column coordinate
users, I = np.unique(UI[:,0], return_inverse=True)
items, J = np.unique(UI[:,1], return_inverse=True)
positives = np.ones(len(UI), int) # points with a positive polarity (where users gave implicit feedback on items)
Po = sparse.coo_matrix((positives, (I, J)))
print(Po); print()
print(Po.A)
assert np.all(Po.A[(I, J)] == 1), "All (I, J) positions should be positive (1)"
from scipy import sparse
# Simulate users rating on items
df_ui = pd.DataFrame([['a', 'abc'],['b', 'def'],['c', 'ghi'],
['d', 'abc'],['a', 'ghi'],['e', 'fg'],
['f', 'f76'],['b', 'f76']],
columns = ['user','item'])
display(df_ui)
UI = df_ui.values
print()
# I: row coordiante, J: column coordinate
users, I = np.unique(UI[:,0], return_inverse=True)
items, J = np.unique(UI[:,1], return_inverse=True)
positives = np.ones(len(UI), int) # points with a positive polarity (where users gave implicit feedback on items)
Po = sparse.coo_matrix((positives, (I, J)))
print(Po); print()
print(Po.A)
assert np.all(Po.A[(I, J)] == 1), "All (I, J) positions should be positive (1)"
| user | item | |
|---|---|---|
| 0 | a | abc |
| 1 | b | def |
| 2 | c | ghi |
| 3 | d | abc |
| 4 | a | ghi |
| 5 | e | fg |
| 6 | f | f76 |
| 7 | b | f76 |
(0, 0) 1 (1, 1) 1 (2, 4) 1 (3, 0) 1 (0, 4) 1 (4, 3) 1 (5, 2) 1 (1, 2) 1 [[1 0 0 0 1] [0 1 1 0 0] [0 0 0 0 1] [1 0 0 0 0] [0 0 0 1 0] [0 0 1 0 0]]