A common practice in modern explainable AI is to post-hoc explain black-box machine learning (ML) predictors – such as neural networks – by extracting symbolic knowledge out of them, in the form of either rule lists or decision trees. By acting as a surrogate model, the extracted knowledge aims at revealing the inner working of the black box, thus enabling its inspection, representation, and explanation.
Various knowledge-extraction algorithms have been presented in the literature so far. Unfortunately, running implementations of most of them are currently either proofs of concept or unavailable. In any case, a unified, coherent software framework supporting them all – as well as their interchange, comparison, and exploitation in arbitrary ML workflows – is currently missing.
Accordingly, in this paper we discuss the design of PSyKE, a platform providing general-purpose support to symbolic knowledge extraction from different sorts of black-box predictors via many extraction algorithms. Notably, PSyKE targets symbolic knowledge in logic form, allowing the extraction of first-order logic clauses. The extracted knowledge is thus both machine- and human-interpretable, and can be used as a starting point for further symbolic processing—e.g. automated reasoning.
AndrewsR., DiederichJ. and TickleA.B., Survey and critique of techniques for extracting rules from trained artificial neural networks, Knowledge-Based Systems8(6) (1995), 373–389.
2.
AzcarragaA., LiuM.D. and SetionoR., Keyword extraction using backpropagation neural networks and rule extraction. In The 2012 International Joint Conference on Neural Networks (IJCNN 2012), pp. 1–7. IEEE, 2012.
3.
BaesensB., SetionoR., De LilleV.,
ViaeneS. and
VanthienenJ., Building credit-risk evaluation expert systems using neural network rule extraction and decision tables. In
StoreyVeda C.,
SarkarSumit, and
DeGrossJanice I., editors, ICIS 2001 Proceedings, pp. 159–168. Association for Information Systems, 2001.
4.
BaesensB., SetionoR., MuesC. and VanthienenJ., Using neural network rule extraction and decision tables for credit-risk evaluation, Management Science49(3) (2003), 312–329.
5.
BarakatN. and DiederichJ., Eclectic rule-extraction from support vector machines, International Journal of Computer and Information Engineering2(5) (2008), 1672–1675.
6.
BolognaG. and PellegriniC., Three medical examples in neural network rule extraction, Physica Medica13 (1997), 183–187.
7.
BoulléM., Khiops: A statistical discretization method of continuous attributes, Machine Learning55(1) (2004), 53–69.
8.
BreimanL., FriedmanJ., StoneC.J. and OlshenR.A., Classification and Regression Trees. CRC Press, 1984.
9.
CalegariR., CiattoG. and OmiciniA., On the integration of symbolic and sub-symbolic techniques for XAI: A survey, Intelligenza Artificiale14(1) (2020), 7–32.
10.
CalvaresiD., CiattoG., NajjarA., AydoğanR.,
Van der TorreL.,
OmiciniA. and
SchumacherM., EXPECTATION: Personalized explainable artificial intelligence for decentralized agents with heterogeneous knowledge. In
CalvaresiDavide,
NajjarAmro,
WinikoffMichael, and
FrämlingKary, editors, Explainable and Transparent AI and Multi-Agent Systems. Third International Workshop, EXTRAAMAS Virtual Event, May 3–7, Revised Selected Papers, volume 12688 of LNCS, pp. 331–343. Springer Nature, Basel, Switzerland, 2021.
11.
CanoA., NguyenD.T., VenturaS. and CiosK.J., ur-caim: improved CAIM discretization for unbalanced and balanced data, Soft Comput20(1) (2016), 173–188.
12.
CiattoG., CalegariR. and OmiciniA., 2PKt: A logic-based ecosystem for symbolic AI, SoftwareX16(100817) (2021), 1–7, December 2021.
13.
CiattoG., CalegariR., OmiciniA. and CalvaresiD., Towards XMAS: eXplainability through Multi-Agent Systems. In
SavaglioClaudio,
FortinoGiancarlo,
CiattoGiovanni, and
OmiciniAndrea, editors, AI&IoT 2019 – Artificial Intelligence and Internet of Things volume 2502 of CEUR Workshop Proceedings, pp. 40–53. CEUR WS, November, 2019.
14.
CiattoG., CalvaresiD., SchumacherM.I. and OmiciniA., An abstract framework for agent-based explanations in AI. In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke-Smith, editors, pp. – IFAAMAS, May, 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1816–1818. IFAAMAS, May 2020.
15.
ColmerauerA., RousselP., The birth of Prolog. In History of programming languages—II, pp. 331–367. ACM, 1996.
16.
CravenM.W. and ShavlikJ.W., Using sampling and queries to extract rules from trained neural networks. In Machine Learning Proceedings1994, pp. 37–45, Elsevier, 1994.
17.
CravenM.W. and ShavlikJ.W., Extracting treestructured representations of trained networks. In David S. Touretzky, Michael C. Mozer, and Michael E. Hasselmo, editors, Advances in Neural Information Processing Systems 8. Proceedings of the 1995 Conference, pp. 24–30. The MIT Press, June 1996.
18.
DoughertyJ., KohaviR. and SahamiM., Supervised and unsupervised discretization of continuous features. In Armand Prieditis and Stuart J. Russell, editors, Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, July 9-12, 1995, pp. 194–202. Morgan Kaufmann, 1995.
19.
ElhilbawiH., EldawlatlyS. and MahdiH., The importance of discretization methods in machine learning applications: A case study of predicting ICU mortality. In Aboul Ella Hassanien, Kuo-Chi Chang, and Mincong Tang, editors, Advanced Machine Learning Technologies and Applications - Proceedings of AMLTA 2021, Cairo, Egypt, March 22-24, 2021, volume 1339 of Advances in Intelligent Systems and Computing, pp. 214–224. Springer, 2021.
20.
FrancoL., Luis SubiratsJ.,
MolinaI.,
AlbaE. and
JerezJ.M., Early breast cancer prognosis prediction and rule extraction using a new constructive neural network algorithm. In Computational and Ambient Intelligence (IWANN 2007), volume 4507 of LNCS, pp. 1004–1011, Springer, 2007.
21.
FreitasAlex A.,
Comprehensible classification models: a position paper, ACM SIGKDD Explorations Newsletter15(1) (2014),1–10, June 2014.
22.
GavanelliM. and RossiF., Constraint logic programming. In Agostino Dovier and Enrico Pontelli, editors, A 25-Year Perspective on Logic Programming: Achievements of the Italian Association for Logic Programming, GULP, volume 6125 of LNCS, pp. 64–86. Springer, 2010.
23.
GuidottiR., MonrealeA., RuggieriS., TuriniF., GiannottiF. and PedreschiD., A survey of methods for explaining black box models, ACM Computing Surveys51(5) (2018), 1–42.
24.
HayashiY., SetionoR. and YoshidaK., A comparison between two neural network rule extraction techniques for the diagnosis of hepatobiliary disorders, Artificial Intelligence in Medicine20(3) (2000), 205–216.
25.
HoK.M.and
Scott Paul D.. Zeta: A global method for discretization of continuous variables. In David Heckerman, Heikki Mannila, and Daryl Pregibon, editors, Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (KDD-97), Newport Beach, California, USA, August 14-17, 1997, pp. 191–194. AAAI Press, 1997.
26.
HofmannA., SchmitzC. and SickB., Rule extraction from neural networks for intrusion detection in computer networks. In 2003 IEEE International Conference on Systems, Man and Cybernetics, volume 2, (2003), pp. 1259–1265. IEEE.
27.
HuysmansJ., BaesensB. and VanthienenJ., ITER: An algorithm for predictive regression rule extraction. In Data Warehousing and Knowledge Discovery (DaWaK 2006), (2006), pp. 270–279. Springer.
28.
HuysmansJ., DejaegerK., MuesC., VanthienenJ. and BaesensB., An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models, Decision Support Systems51(1) (2011), 141–154.
29.
JaffarJ. and MaherM.J., Constraint logic programming: A survey, Journal of Logic Programming 19/20 (1994), 503–581.
30.
KennyE.M., FordC., QuinnM. and KeaneM.T., Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and errorrates in XAI user studies, Artificial Intelligence294 (2021), 103459.
31.
KerberR., Chimerge: Discretization of numeric attributes. In William R. Swartout, editor, Proceedings of the 10th National Conference on Artificial Intelligence, San Jose, CA, USA, July 12-16, 1992, pp. 123–128. AAAI Press / The MIT Press, 1992.
32.
KonigR., JohanssonU. and NiklassonL., G-REX: A versatile framework for evolutionary data mining. In 2008 IEEE International Conference on Data Mining Workshops (ICDM 2008 Workshops) (2008), pp. 971–974.
33.
KurganL.A. and CiosK.J., CAIM discretization algorithm, IEEE Transactions on Knowledge and Data Engineering16(2) (2004), 145–153.
34.
LiptonZ.C., The mythos of model interpretability, Queue16(3) (2018),31–57, June 2018.
35.
John W. Lloyd, editor. Computational Logic. Springer, 1990.
36.
MartensD., BaesensB., GestelT.V. and VanthienenJ., Comprehensible credit scoring models using rule extraction from support vector machines, European Journal of Operational Research183(3) (2007), 1466–1476.
37.
MetakidesG. and NerodeA., Principles of logic and logic programming, volume 13 of Studies in Computer Science and Artificial Intelligence, Elsevier, 1996.
38.
MurphyP.M. and PazzaniM.J., Id2-of-3: Constructive induction of m-of-n concepts for discriminators in decision trees. In Machine Learning Proceedings (1991), pp. 183–187, Elsevier.
39.
PedregosaF., VaroquauxG., GramfortA., MichelV., ThirionB., GriselO., BlondelM., PrettenhoferP., WeissR., DubourgV., VanderPlasJ., PassosA., CournapeauD., BrucherM., PerrotM. and DuchesnayE., Scikitlearn: Machine learning in Python, Journal of Machine Learning Research (JMLR)12 (2011), 2825–2830.
40.
Ross QuinlanJ., Simplifying decision trees, International Journal of Man-Machine Studies27(3) (1987), 221–234.
41.
RossJ., Quinlan, C4.5: Programming for machine learning. Morgan Kauffmann, 1993.
42.
RochaA., Paulo PapaJ. and
MeiraLuis A. A., How far do we get using machine learning black-boxes?International Journal of Pattern Recognition and Artificial Intelligence26(02) (2012), 1261001–(1–23).
43.
RudinC., Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence1(5) (2019), 206–215.
44.
SabbatiniF., CiattoG., CalegariR. and OmiciniA., On the design of PSyKE: A platform for symbolic knowledge extraction. In Roberta Calegari, Giovanni Ciatto, Enrico Denti, Andrea Omicini, and Giovanni Sartor, editors, WOA 2021 – 22nd Workshop “From Objects to Agents”, volume of CEUR Workshop Proceedings, pp. 29–48. Sun SITE Central Europe, RWTH Aachen University, October 2021. 22ndWorkshop “From Objects to Agents” (WOA 2021), Bologna, Italy, 1–3 September 2021. Proceedings.
45.
SabbatiniF., CiattoG. and OmiciniA., GridEx: An algorithm for knowledge extraction from black-box regressors. In Davide Calvaresi, Amro Najjar, Michael Winikoff, and Kary Främling, editors, Explainable and Transparent AI and Multi-Agent Systems. Third International Workshop, EXTRAAMAS 2021, Virtual Event, May 3–7, 2021, Revised Selected Papers, volume 12688 of LNCS, pp. 18– 38. Springer Nature, Basel, Switzerland, 2021.
46.
SaitoK. and NakanoR., Extracting regression rules from neural networks, Neural Networks15(10) (2002), 1279–1288.
47.
Gregor P. J. Schmitz, C. Aldrich and François S. Gouws,
ANN-DT: an algorithm for extraction of decision trees from artificial neural networks, IEEE Transactions on Neural Networks10(6) (1999), 1392–1401.
48.
SetionoR., BaesensB. and MuesC., Rule extraction from minimal neural networks for credit card screening, International Journal of Neural Systems21(04) (2011), 265–276.
49.
SetionoR., Kheng LeowW. and
Zurada JacekM., Extraction of rules from artificial neural networks for nonlinear regression, IEEE Transactions on Neural Networks13(3) (2002), 564–577.
50.
Smile. Statistical machine intelligence and learning engine. https://haifengl.github.io. [Online; last accessed 13 Jan 2022].
51.
SteinerM.T.A., NetoP.J.S., SomaN.Y., ShimizuT. and NievolaJ.C., Using neural network rule extraction for credit-risk evaluation, International Journal of Computer Science and Network Security6(5A) (2006), 6–16.
52.
TowellG.G.and
ShavlikJude W., Extracting refined rules from knowledge-based neural networks, Machine Learning13(1) (1993), 71–101.
53.
YangY., WebbG.I. and WuX., Discretization methods. In Oded Maimon and Lior Rokach, editors, Data Mining and Knowledge Discovery Handbook, 2nd ed, pp. 101–116. Springer, 2010.
