publications
You can also check my Google Scholar profile.
2026
- JARVIS-Bench: Benchmarking Personal Intelligence Agents on Long-Horizon Real-User Daily TracesWeizhi Zhang, Wei-Chieh Huang, Yueqing Liang, Liwei Jiang, Zhengxiang Wang, Liangwei Yang , and 53 more authorsIn Advances in Neural Information Processing Systems . Also accepted at the Human-AI Coevolution (HAIC) workshop , Dec 2026Accepted for poster presentation
Personal intelligence agents aim to support users across daily-life tasks by learning their preferences, routines, needs, and emotional states from long-term interaction. They are expected to anticipate user needs, provide proactive assistance, and infer preferences that users have never explicitly articulated. However, existing benchmarks for LLM personalization lack real user data spanning extended periods and diverse daily-life domains, and typically reduce personalization to simplified settings such as recommendation or fact retrieval. We introduce JARVIS-Bench, a new benchmark for evaluating personal intelligence agents, curated from real users’ daily traces and augmented with synthetic trajectory expansions grounded in implicit-preference insights from surveys and structured personas. JARVIS-Bench is built from participants who contributed 28 to 44 days of multimodal self-tracking logs, structured personas, and a 100-question implicit-preference survey spanning eight domains. It supports four downstream task surfaces: implicit preference reasoning, proactive-help prediction, emotion-aware modeling, and cross-domain transfer, all evaluated through a unified event iterator and long-horizon user-memory pipeline. We benchmark 28 LLMs from 10 model families, including Gemini, GPT, Qwen, Gemma, and Claude, across a five-condition input ladder ranging from no memory to lifelong memory. Results show that memory and model scaling improve personalization, yet current frontier models remain far from achieving robust real-world personal intelligence.
@inproceedings{zhang_jarvis-bench_2026, address = {Atlanta, US}, title = {{JARVIS}-{Bench}: {Benchmarking} {Personal} {Intelligence} {Agents} on {Long}-{Horizon} {Real}-{User} {Daily} {Traces}}, volume = {39}, shorttitle = {{JARVIS}-{Bench}}, url = {https://openreview.net/forum?id=ACYXGndPs2}, language = {English}, booktitle = {Advances in {Neural} {Information} {Processing} {Systems}}, publisher = {Curran Associates, Inc.}, author = {Zhang, Weizhi and Huang, Wei-Chieh and Liang, Yueqing and Jiang, Liwei and Wang, Zhengxiang and Yang, Liangwei and Li, Zechen and Wu, Yuchen and Zhang, Haozhen and Wang, Yu and Bei, Yuanchen and Zhou, Yue and Zhu, Siqi and Zou, Henry Peng and Liu, Jiahong and Zhang, Xinni and Martin, Paul and Imperial, Joseph Marvin and Luo, Yuyang and Xu, Xiongxiao and Huang, Baixiang and Wu, Shanglin and Resck, Lucas and Wang, Yibo and Liu, Yuqing and He, Langzhou and Li, Chengze and Tian, Yuxin and Zheng, Kening and Ma, Enze and Huynh, Boi and Hui, Zheng and Yang, Rui and Feng, Tao and Yang, Jie and Li, Shanghao and Wang, Haoran and Yang, Wooseong and Zhang, Hanrong and Ma, Huanhuan and Yoon, Daye and Wu, Yaozu and Lian, Shikan and Ruan, Xiaoqian and Wang, Fangxin and Park, Hyeonjeong and Hu, Hins and Fan, Wenzhe and Wang, Chen and Gu, Yifan and Li, Dongyuan and Wang, Song and Caliskan, Aylin and Wang, Jindong and Luo, Xiao and Chen, Yankai and Shu, Kai and Liu, Xue and Yu, Philip S.}, month = dec, year = {2026}, note = {Accepted for poster presentation}, bibtex_show = true, google_scholar_id = {Wp0gIr-vW9MC}, } - CulturalRiddles: A Multicultural Benchmark of RiddlesTianyi Hu, Henry Gagnier, Vinod Anbalagan, My Chiffon Nguyen, Pouya Sadeghi, Farah Abdou , and 133 more authorsIn 6th Multilingual Representation Learning (MRL) Workshop 2026 at EMNLP , Oct 2026Non-archival extended abstract
Capturing how well LLMs can understand and model the diversity of cultures around the globe has increasingly gained importance as LLMs’ language coverage has rapidly advanced. However, existing benchmarks are still knowledge- and English-centric with limited coverage and complexity. We propose CulturalRiddles, a benchmark of culturally-grounded riddles, spanning 61 cultures and 51 languages, created in a participatory community effort. These riddles are both hyper-specific to each culture, require factual knowledge, language skills, social knowledge, and strong abductive reasoning skills to solve. We benchmark 28 LLMs with both automatic and human evaluation and release all artifacts publicly.
@inproceedings{hu_culturalriddles_2026, address = {Budapest, Hungary}, title = {{CulturalRiddles}: {A} {Multicultural} {Benchmark} of {Riddles}}, shorttitle = {{CulturalRiddles}}, language = {English}, author = {Hu, Tianyi and Gagnier, Henry and Anbalagan, Vinod and Nguyen, My Chiffon and Sadeghi, Pouya and Abdou, Farah and Mekki, Abdellah EL and Aziz, Ahanaf and Ramadhan, Ahmad mufli and Eppa, Akhil and Gimei, Alex and Nikitina, Anastasiia and Maity, Ankita and Hariharan, Arya and Suneesh, Arya and Naebzadeh, Aylin and Munot, Ayush Sunil and Chen, Bryan and Hoang, Christian and Mo, Di and Kozachek, Diana and Pisarevskaya, Dina and Serrano-Miranda, Diogo M. and Poudel, Divas and Yukhymenko, Hanna and Purbey, Jebish and Zheng, Jingyi and Swindell, Jonathan and Sun, Kaiser and Sunkara, Lakshmi Tejaswi and Navarro, Luis Fernando SOLIS and Smith, Madeline and Apu, Md Rabius Sany and Altomare, Micol and Ajayi, Oluwaseun A. and Rizova, Ralitza Nikolova and Kadiyala, Ram Mohan Rao and Elframawy, Rawan Mohammed and Dhar, Ruchira and Amir, Shayan and Fernandez, Silvia Andrea and Sunkara, Venkata Manaswitha and Silimkhan, Vivek and Ghussin, Yusser Al and Al-Khalili, Zena and Issam, Abderrahmane and Assadi, Adnan El and Sant, Aleix and Borah, Angana and Suresh, Ananth K. and Sahu, Ananya and Hu, Bin and Balcha, Bontu Fufa and Clemente, Borja Sánchez and Lothritz, Cedric and Liu, Chen Cecilia and Jokić, Danka and Stap, David and Yadeta, Debela Desalegn and Poudel, Dikshya and Krzemiński, Dominik and Sánchez, Eduardo and Anand, Emile Timothy and Su, Emily Z. and Adenuga, Esther and Bochini, Felipe and Mo, Fengran and Philippy, Fred and Assis, Gabriel and Li, Haichang and Mehrafarin, Houman and Afanasev, Ilia and Shihab, Istiak and Nayem, Jannatul and Gilabert, Javier García and Chundru, Jayanth Krishna and Zhang, Jinghan and Radoła, Joanna and Avalos, José Matías and Olusanya, Joy and Carvalho, Juliana and Huang, Kaiyu and Dhole, Kaustubh and Altomare, Lauren and Cossio, Leandro Andrés and Wang, Lingbo and Alazraki, Lisa and Resck, Lucas and Pàmies, Marc and Santis, Marco De and Mofakhami, Mehrnaz and Zhang, Mike and Habibi, Muhammad Ravi Shulthan and Umar, Muhammad Saad and Elframawy, Nadine and Bhatia, Naman and Chen, Nancy F. and Tasnim, Nazia and Bafna, Niyati and Sarwar, Nobin and Volchek, Oksana and Bugaiova, Olena and Gibert, Ona de and Triamamornwooth, Patteera and Delgado, Pedro and Movva, Prahitha and Rane, Prerana and Fernandes, Reuben Chagas and Choenni, Rochelle and Lee, Roy Ka-Wei and Tan, Rui Yang and Zawar, Rushikesh and Gade, Saanidhi and Swami, Sajag and Singaraju, Santhoshini Pranathi and Rajaee, Sara and Bahrami, Sarah and Bopitiya, Sayuru Rehan and Venkitachalam, Shankar and Dipta, Shubhashis Roy and Avila, Silvia Lorena and Bhattacharyya, Souvik and Chen, Stephy and Rajwal, Swati and Bose, Tulika and Ngo, Van and Kumar, Vanya Bannihatti and Poritski, Vladislav and Aissa, Wafa and Zheng, Weihua and El-Sayed, Wesam and Bagla, Yash and Paniv, Yurii and Baannou, Zakaria and Liu, Zhengyuan and Fadaee, Marzieh and Arora, Akhil and Boczka, Károly and Kreutzer, Julia}, month = oct, year = {2026}, note = {Non-archival extended abstract}, booktitle = {6th Multilingual Representation Learning (MRL) Workshop 2026 at {EMNLP}}, bibtex_show = true, google_scholar_id = {4DMP91E08xMC} } - The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level FallsLucas Resck, Jingyi Sun, Qianli Wang, Isabelle Augenstein, and Anna KorhonenarXiv , Oct 2026
Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning. To our knowledge, we are the first to relate CoT faithfulness to both resource level and task type (knowledge- or reasoning-intensive). On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures. We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks. Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap. Languages in which models are less confident rely more on reasoning, yet less confident answers within a language do not, a Simpson’s paradox. In lower-resource languages, models rely on their reasoning for more answers, not more strongly for each. CoT monitoring may therefore be most informative in lower-resource languages, where it is least often validated.
@misc{resck_reliance_2026, title = {The {Reliance} {Hypothesis}: {Chain}-of-{Thought} {Faithfulness} {Rises} as {Language} {Resource} {Level} {Falls}}, copyright = {Creative Commons Attribution 4.0 International}, shorttitle = {The {Reliance} {Hypothesis}}, url = {http://arxiv.org/abs/2610.22841}, doi = {10.48550/arXiv.2610.22841}, publisher = {arXiv}, author = {Resck, Lucas and Sun, Jingyi and Wang, Qianli and Augenstein, Isabelle and Korhonen, Anna}, month = oct, year = {2026}, bibtex_show = true, google_scholar_id = {mVmsd5A6BfQC}, } - Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference OptimisationLucas Resck, Isabelle Augenstein, and Anna KorhonenarXiv , Oct 2026
Multilingual large language models (MLLMs) often answer culturally ambiguous questions inconsistently across languages, even for a fixed user persona – e.g., Shakespeare in English but Cervantes in Spanish for the same British user. To robustly quantify this Cross-lingual Cultural Inconsistency, we introduce Singleton Fleiss’s Kappa, a metric mathematically resilient to hallucinations. For mitigation, we propose Cross-lingual Cultural Consistent Preference Optimisation (C-3PO), a consensus-driven alignment framework. C-3PO raises Kappa by 0.07–0.10 over unaligned models, above every prompting and representation steering baseline averaged across personas, whilst largely preserving explicit user identities, cultural neutrality and intrinsic cultural knowledge; where it does not, a soft-consensus variant recovers most of the cost. Although trained only on multiple-choice questions, C-3PO also improves consistency in open-ended generation. Empirical evaluations demonstrate this inconsistency is more pronounced for lower-resource languages like Indonesian and Persian. Finally, early decoding of intermediate layers reveals that MLLMs implicitly personalise outputs towards the prompt language’s stereotypical culture as forward-pass representations stabilise.
@misc{resck_mitigating_2026, title = {Mitigating {Cross}-{Lingual} {Cultural} {Inconsistencies} in {LLMs} via {Consensus}-{Driven} {Preference} {Optimisation}}, copyright = {Creative Commons Attribution 4.0 International}, url = {http://arxiv.org/abs/2605.12515}, doi = {10.48550/arXiv.2605.12515}, publisher = {arXiv}, author = {Resck, Lucas and Augenstein, Isabelle and Korhonen, Anna}, month = oct, year = {2026}, bibtex_show = true, google_scholar_id = {M3ejUd6NZC8C}, } - Lower-Resource, Higher Scores: Language Bias in LLM EvaluatorsEj Zhou, Lucas Resck, Zheng Hui*, and Anna Korhonen*In The Third Annual Conference on Language Modeling , Oct 2026
LLM evaluators (trained reward models and prompted LLM-as-a-Judge) are routinely validated via pairwise accuracy. In a multilingual setting, this operates under the premise that high pairwise accuracy implies reliable, language-neutral scoring. We show that this assumption does not hold across languages. We conduct experiments with semantically identical instruction–response pairs across 23 languages, and find that multilingual evaluators assign significantly different scores to different evaluation languages. We show that this bias is statistically significant and consistent across eight evaluators of different architectures and training paradigms, and is strongly correlated with language resource level: lower-resource languages are scored more generously. Meanwhile, these biases are invisible to pairwise accuracy: evaluators would achieve above 90% pairwise accuracy, yet have 43% difference in acceptance-rate across languages under a global decision threshold, meaning, for instance, that harmful content in lower-resource languages is more likely to pass safety filters. We then investigate why lower-resource languages receive higher rather than lower scores, and we find that model uncertainty (measured by negative log-likelihood) is linked with the effect: models tend to give higher scores when less confident; however, language identity remains a significant predictor after controlling for uncertainty, and the bias cannot be explained away by content difficulty alone, but is a structural, language-level misalignment.
@inproceedings{zhou_lower-resource_2026, title = {<i>Lower-{Resource}, {Higher} {Scores}</i>: {Language} {Bias} in {LLM} {Evaluators}}, copyright = {Creative Commons Attribution 4.0 International}, shorttitle = {Lower-{Resource}, {Higher} {Scores}}, url = {http://arxiv.org/abs/2607.14480}, doi = {10.48550/arXiv.2607.14480}, booktitle = {The {Third} {Annual} {Conference} on {Language} {Modeling}}, author = {Zhou, Ej and Resck, Lucas and Hui, Zheng and Korhonen, Anna}, month = oct, year = {2026}, bibtex_show = true, google_scholar_id = {aqlVkmm33-oC}, } - LegalAnalytics: bridging visual explanations and workload streamline in Brazilian Supreme Court appealsLucas Resck, Felipe Moreno-Vera, Tobias Veiga, Gerardo Paucar, Ezequiel Fajreldines, Guilherme Klafke , and 2 more authorsArtificial Intelligence and Law, Sep 2026
The Brazilian Supreme Court serves as the highest judicial authority in Brazil and is responsible for adjudicating constitutional matters presented as extraordinary appeals. These appeals undergo a rigorous screening process guided by established legal principles known as Topics of General Repercussion. Seeking to streamline this procedure, we developed LegalAnalytics to explore the research question: Can machine learning and explainable AI techniques enhance the classification of appeals in legal workflows? LegalAnalytics harnesses advanced natural language processing algorithms and classification models to categorize each appeal according to the most pertinent topics accurately. In addition, it incorporates LIME (Local Interpretable Model-agnostic Explanations) to highlight the key sections of an appeal and compare them with relevant precedents. This approach ensures a transparent justification for every classification. The system is thoughtfully designed with a user-friendly interface tailored for public servants, judges, and lawyers. Extensive testing with dozens of legal experts confirmed the effectiveness of LegalAnalytics, with consistently positive feedback underscoring its significant practical impact.
@article{resck_legalanalytics_2026, title = {{LegalAnalytics}: bridging visual explanations and workload streamline in {Brazilian} {Supreme} {Court} appeals}, volume = {34}, copyright = {Copyright © 2025 The Authors. Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).}, issn = {1572-8382}, shorttitle = {{LegalAnalytics}}, url = {https://doi.org/10.1007/s10506-025-09446-w}, doi = {10.1007/s10506-025-09446-w}, language = {en}, number = {3}, journal = {Artificial Intelligence and Law}, publisher = {Springer Nature}, author = {Resck, Lucas and Moreno-Vera, Felipe and Veiga, Tobias and Paucar, Gerardo and Fajreldines, Ezequiel and Klafke, Guilherme and Nonato, Luis G. and Poco, Jorge}, month = sep, year = {2026}, pages = {685--743}, bibtex_show = true, google_scholar_id = {kNdYIx-mwKoC} } - Language-Dependent Miscalibration in Multilingual LLM EvaluatorsEj Zhou, Lucas Resck, Zheng Hui, and Anna KorhonenIn I Can’t Believe It’s Not Better: Where Large Language Models Need to Improve (ICBINB 2026) at ICLR , Apr 2026Non-archival
Prompted LLM-as-a-Judge systems or trained reward models are typically validated using pairwise accuracy, under the assumption that high accuracy implies reliable and language-invariant evaluation. We demonstrate that multilingual LLM evaluators exhibit large, systematic, and statistically significant language-dependent bias in pointwise scoring. We show that this mismatch has concrete downstream consequences: threshold filtering can result in huge differences in acceptance rates.
@inproceedings{zhou_language-dependent_2026, address = {Rio de Janeiro, Brazil}, title = {Language-{Dependent} {Miscalibration} in {Multilingual} {LLM} {Evaluators}}, url = {https://openreview.net/forum?id=vvnTA1aReN}, language = {en}, author = {Zhou, Ej and Resck, Lucas and Hui, Zheng and Korhonen, Anna}, month = apr, year = {2026}, note = {Non-archival}, booktitle = {I Can't Believe It's Not Better: Where Large Language Models Need to Improve (ICBINB 2026) at {ICLR}}, bibtex_show = true, google_scholar_id = {4TOpqqG69KYC} } - Glints of Gold or Troubling Waters? Can a School of Merged Monolingual Goldfish Models Swim in Bilingual Seas?Suchir Salhan, Ej Zhou*, Laura Barbenel*, Aoife O’Driscoll*, Lily Goulder*, Lucas Resck , and 2 more authorsIn First Workshop on Multilingual Multicultural Evaluation (MME 2026) at EACL , Mar 2026Non-archival
@inproceedings{salhan_glints_2026, address = {Rabat, Morocco}, title = {<i>Glints of {Gold} or {Troubling} {Waters}?</i> {Can} a {School} of {Merged} {Monolingual} {Goldfish} {Models} {Swim} in {Bilingual} {Seas}?}, shorttitle = {Glints of {Gold} or {Troubling} {Waters}?}, language = {en}, author = {Salhan, Suchir and Zhou, Ej and Barbenel, Laura and O’Driscoll, Aoife and Goulder, Lily and Resck, Lucas and Arnett, Catherine and Buttery, Paula}, month = mar, year = {2026}, note = {Non-archival}, booktitle = {First Workshop on Multilingual Multicultural Evaluation (MME 2026) at {EACL}}, bibtex_show = true, google_scholar_id = {_kc_bZDykSQC} }
2025
- Explainability and Interpretability of Multilingual Large Language Models: A SurveyLucas Resck, Isabelle Augenstein, and Anna KorhonenIn Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , Nov 2025
Multilingual large language models (MLLMs) demonstrate state-of-the-art capabilities across diverse cross-lingual and multilingual tasks. Their complex internal mechanisms, however, often lack transparency, posing significant challenges in elucidating their internal processing of multilingualism, cross-lingual transfer dynamics and handling of language-specific features. This paper addresses this critical gap by presenting a survey of current explainability and interpretability methods specifically for MLLMs. To our knowledge, it is the first comprehensive review of its kind. Existing literature is categorised according to the explainability techniques employed, the multilingual tasks addressed, the languages investigated and available resources. The survey further identifies key challenges, distils core findings and outlines promising avenues for future research within this rapidly evolving domain.
@inproceedings{resck_explainability_2025, address = {Suzhou, China}, title = {Explainability and {Interpretability} of {Multilingual} {Large} {Language} {Models}: {A} {Survey}}, copyright = {Copyright © 2025 Association for Computational Linguists (ACL). All Rights Reserved. Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).}, isbn = {979-8-89176-332-6}, shorttitle = {Explainability and {Interpretability} of {Multilingual} {Large} {Language} {Models}}, url = {https://aclanthology.org/2025.emnlp-main.1033/}, doi = {10.18653/v1/2025.emnlp-main.1033}, language = {English}, booktitle = {Proceedings of the 2025 {Conference} on {Empirical} {Methods} in {Natural} {Language} {Processing}}, publisher = {Association for Computational Linguistics}, author = {Resck, Lucas and Augenstein, Isabelle and Korhonen, Anna}, editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet}, month = nov, year = {2025}, pages = {20465--20497}, bibtex_show = true, google_scholar_id = {Zph67rFs4hoC}, } - Dark Chest of Wonders: On the Specific Language-Specific Features of Multilingual Large Language ModelsLucas Resck*, and Ej Zhou*In 5th Workshop on Multilingual Representation Learning (MRL 2025) at EMNLP , Nov 2025Non-archival extended abstract
@inproceedings{resck_dark_2025, address = {Suzhou, China}, title = {Dark {Chest} of {Wonders}: {On} the {Specific} {Language}-{Specific} {Features} of {Multilingual} {Large} {Language} {Models}}, shorttitle = {Dark {Chest} of {Wonders}}, language = {English}, author = {Resck, Lucas and Zhou, Ej}, month = nov, year = {2025}, note = {Non-archival extended abstract}, booktitle = {5th Workshop on Multilingual Representation Learning (MRL 2025) at {EMNLP}}, bibtex_show = true, google_scholar_id = {ULOm3_A8WrAC} } - Explainable NLLP: Advancements in Explainable AI for Natural Legal Language ProcessingLucas Resck, Felipe Moreno-Vera, Tobias Veiga, Gerardo Paucar, Ezequiel Fajreldines, Guilherme Klafke , and 2 more authorsIn Proceedings of the Seventh Workshop on Automated Semantic Analysis of Information in Legal Texts co-located with the 20th International Conference on Artificial Intelligence and Law (ICAIL 2025) , Jun 2025Short paper
Despite the increasing application of machine learning and NLP methods in the legal domain, there has been limited effort to enhance the understanding and transparency of these algorithms. This paper addresses this gap by presenting a survey on Explainable AI (XAI) applied to Natural Legal Language Processing (NLLP). To our knowledge, this survey represents the first comprehensive examination at the intersection of XAI, Law, and NLP. Building upon prior surveys focused on partial intersections of these domains, we propose a taxonomy for classifying papers based on the NLLP task, explanation type, and technique employed. Additionally, we delve into discussions surrounding Explainable NLLP, considering perspectives related to ethics, current open issues, and future work. Our analysis reveals that the categorized papers generally do not thoroughly examine the ethical implications of the explainability principle in NLP within the legal field. Furthermore, they do not discuss the role and value of explanations nor do they effectively utilize their respective XAI techniques to offer insights into the limitations of NLP systems.
@inproceedings{resck_explainable_2025, address = {Chicago, USA}, title = {Explainable {NLLP}: {Advancements} in {Explainable} {AI} for {Natural} {Legal} {Language} {Processing}}, volume = {4174}, copyright = {© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/deed.en).}, shorttitle = {Explainable {NLLP}}, url = {https://ceur-ws.org/Vol-4174/paper6.pdf}, language = {English}, booktitle = {Proceedings of the {Seventh} {Workshop} on {Automated} {Semantic} {Analysis} of {Information} in {Legal} {Texts} co-located with the 20th {International} {Conference} on {Artificial} {Intelligence} and {Law} ({ICAIL} 2025)}, publisher = {CEUR Workshop Proceedings}, author = {Resck, Lucas and Moreno-Vera, Felipe and Veiga, Tobias and Paucar, Gerardo and Fajreldines, Ezequiel and Klafke, Guilherme and Nonato, Luis Gustavo and Poco, Jorge}, month = jun, year = {2025}, note = {Short paper}, pages = {69--82}, bibtex_show = true, google_scholar_id = {YOwf2qJgpHMC} } - Empirical analysis of binding precedent efficiency in Brazilian Supreme Court via case classificationRaphaël Tinarrage, Henrique Ennes, Lucas Resck, Lucas T. Gomes, Jean R. Ponciano, and Jorge PocoArtificial Intelligence and Law, May 2025
Binding precedents (súmulas vinculantes) constitute a juridical instrument unique to the Brazilian legal system and whose objectives include the protection of the Federal Supreme Court against repetitive demands. Studies of the effectiveness of these instruments in decreasing the Court’s exposure to similar cases, however, indicate that they tend to fail in such a direction, with some of the binding precedents seemingly creating new demands. We empirically assess the legal impact of five binding precedents, 11, 14, 17, 26, and 37, at the highest Court level through their effects on the legal subjects they address. This analysis is only possible through the comparison of the Court’s ruling about the precedents’ themes before they are created, which means that these decisions should be detected through techniques of Similar Case Retrieval, which we tackle from the angle of Case Classification. The contributions of this article are therefore twofold: on the mathematical side, we compare the use of different methods of Natural Language Processing — TF-IDF, LSTM, Longformer, and regex — for Case Classification, whereas on the legal side, we contrast the inefficiency of these binding precedents with a set of hypotheses that may justify their repeated usage. We observe that the TF-IDF models performed slightly better than LSTM and Longformer when compared through common metrics; however, the deep learning models were able to detect certain important legal events that TF-IDF missed. On the legal side, we argue that the reasons for binding precedents to fail in responding to repetitive demand are heterogeneous and case-dependent, making it impossible to single out a specific cause. We identify five main hypotheses, which are found in different combinations in each of the precedents studied.
@article{tinarrage_empirical_2025, title = {Empirical analysis of binding precedent efficiency in {Brazilian} {Supreme} {Court} via case classification}, copyright = {Creative Commons CC BY 4.0. Copyright vested in the Author(s).}, issn = {1572-8382}, url = {https://doi.org/10.1007/s10506-025-09458-6}, doi = {10.1007/s10506-025-09458-6}, language = {en}, journal = {Artificial Intelligence and Law}, author = {Tinarrage, Raphaël and Ennes, Henrique and Resck, Lucas and Gomes, Lucas T. and Ponciano, Jean R. and Poco, Jorge}, month = may, year = {2025}, bibtex_show = true, google_scholar_id = {3fE2CSJIrl8C}, }
2024
- Exploring the Trade-off Between Model Performance and Explanation Plausibility of Text Classifiers Using Human RationalesLucas Resck, Marcos M. Raimundo, and Jorge PocoIn Findings of the Association for Computational Linguistics: NAACL 2024 . Also presented as a poster at the LatinX in NLP at NAACL 2024 workshop , Jun 2024
Saliency post-hoc explainability methods are important tools for understanding increasingly complex NLP models. While these methods can reflect the model’s reasoning, they may not align with human intuition, making the explanations not plausible. In this work, we present a methodology for incorporating rationales, which are text annotations explaining human decisions, into text classification models. This incorporation enhances the plausibility of post-hoc explanations while preserving their faithfulness. Our approach is agnostic to model architectures and explainability methods. We introduce the rationales during model training by augmenting the standard cross-entropy loss with a novel loss function inspired by contrastive learning. By leveraging a multi-objective optimization algorithm, we explore the trade-off between the two loss functions and generate a Pareto-optimal frontier of models that balance performance and plausibility. Through extensive experiments involving diverse models, datasets, and explainability methods, we demonstrate that our approach significantly enhances the quality of model explanations without causing substantial (sometimes negligible) degradation in the original model’s performance.
@inproceedings{resck_exploring_2024, address = {Mexico City, Mexico}, title = {Exploring the {Trade}-off {Between} {Model} {Performance} and {Explanation} {Plausibility} of {Text} {Classifiers} {Using} {Human} {Rationales}}, url = {https://aclanthology.org/2024.findings-naacl.262}, doi = {10.18653/v1/2024.findings-naacl.262}, language = {English}, booktitle = {Findings of the {Association} for {Computational} {Linguistics}: {NAACL} 2024}, publisher = {Association for Computational Linguistics}, author = {Resck, Lucas and Raimundo, Marcos M. and Poco, Jorge}, editor = {Duh, Kevin and Gomez, Helena and Bethard, Steven}, month = jun, year = {2024}, pages = {4190--4216}, bibtex_show = true, google_scholar_id = {5nxA0vEk-isC}, } - MSc thesis
Balancing performance and explanation plausibility: a multi-objective approach to text classification with human rationalesLucas Emanuel Resck DominguesFundação Getulio Vargas , May 2024Saliency post-hoc explainability methods are important tools for understanding increasingly complex NLP models. While these methods can reflect the model’s reasoning, they may not align with human intuition, making the explanations not plausible. In this work, we present a methodology for incorporating rationales, which are text annotations explaining human decisions, into text classification models. This incorporation enhances the plausibility of post-hoc explanations while preserving their faithfulness. Our approach is agnostic to model architectures and explainability methods. We introduce the rationales during model training by augmenting the standard cross-entropy loss with a novel loss function inspired by contrastive learning. By leveraging a multi-objective optimization algorithm, we explore the trade-off between the two loss functions and generate a Pareto-optimal frontier of models that balance performance and plausibility. Through extensive experiments involving diverse models, datasets, and explainability methods, we demonstrate that our approach significantly enhances the quality of model explanations without causing substantial (sometimes negligible) degradation in the original model’s performance.
@mastersthesis{domingues_balancing_2024, address = {Rio de Janeiro, Brazil}, title = {Balancing performance and explanation plausibility: a multi-objective approach to text classification with human rationales}, shorttitle = {Balancing performance and explanation plausibility}, url = {https://hdl.handle.net/10438/35362}, language = {eng}, school = {Fundação Getulio Vargas}, author = {Domingues, Lucas Emanuel Resck}, month = may, year = {2024}, bibtex_show = true, google_scholar_id = {MXK_kJrjxJIC}, }
2023
- LegalVis: Exploring and Inferring Precedent Citations in Legal DocumentsLucas E. Resck*, Jean R. Ponciano*, Luis Gustavo Nonato, and Jorge PocoIEEE Transactions on Visualization and Computer Graphics. Presented at Visualization & Visual Analytics (VIS) 2022 , Jun 2023
To reduce the number of pending cases and conflicting rulings in the Brazilian Judiciary, the National Congress amended the Constitution, allowing the Brazilian Supreme Court (STF) to create binding precedents (BPs), i.e., a set of understandings that both Executive and lower Judiciary branches must follow. The STF’s justices frequently cite the 58 existing BPs in their decisions, and it is of primary relevance that judicial experts could identify and analyze such citations. To assist in this problem, we propose LegalVis, a web-based visual analytics system designed to support the analysis of legal documents that cite or could potentially cite a BP. We model the problem of identifying potential citations (i.e., non-explicit) as a classification problem. However, a simple score is not enough to explain the results; that is why we use an interpretability machine learning method to explain the reason behind each identified citation. For a compelling visual exploration of documents and BPs, LegalVis comprises three interactive visual components: the first presents an overview of the data showing temporal patterns, the second allows filtering and grouping relevant documents by topic, and the last one shows a document’s text aiming to interpret the model’s output by pointing out which paragraphs are likely to mention the BP, even if not explicitly specified. We evaluated our identification model and obtained an accuracy of 96%; we also made a quantitative and qualitative analysis of the results. The usefulness and effectiveness of LegalVis were evaluated through two usage scenarios and feedback from six domain experts.
@article{resck_legalvis_2023, title = {{LegalVis}: {Exploring} and {Inferring} {Precedent} {Citations} in {Legal} {Documents}}, volume = {29}, issn = {1941-0506}, shorttitle = {{LegalVis}}, url = {https://ieeexplore.ieee.org/document/9716779/}, doi = {10.1109/TVCG.2022.3152450}, language = {English}, number = {6}, journal = {IEEE Transactions on Visualization and Computer Graphics}, author = {Resck, Lucas E. and Ponciano, Jean R. and Nonato, Luis Gustavo and Poco, Jorge}, month = jun, year = {2023}, keywords = {Data visualization, Task analysis, Visual analytics, Law, Natural language processing, Legal documents, natural language processing, Analytical models, Text analysis, Brazilian legal system, visual analytics}, pages = {3105--3120}, bibtex_show = true, google_scholar_id = {ufrVoPGSRksC}, } - Distill n’ Explain: explaining graph neural networks using simple surrogatesTamara Pereira, Erik Nascimento, Lucas E. Resck, Diego Mesquita, and Amauri SouzaIn Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , Apr 2023
Explaining node predictions in graph neural networks (GNNs) often boils down to finding graph substructures that preserve predictions. Finding these structures usually implies back-propagating through the GNN, bonding the complexity (e.g., number of layers) of the GNN to the cost of explaining it. This naturally begs the question: Can we break this bond by explaining a simpler surrogate GNN? To answer the question, we propose Distill n’ Explain (DnX). First, DnX learns a surrogate GNN via knowledge distillation. Then, DnX extracts node or edge-level explanations by solving a simple convex program. We also propose FastDnX, a faster version of DnX that leverages the linear decomposition of our surrogate model. Experiments show that DnX and FastDnX often outperform state-of-the-art GNN explainers while being orders of magnitude faster. Additionally, we support our empirical findings with theoretical results linking the quality of the surrogate model (i.e., distillation error) to the faithfulness of explanations.
@inproceedings{pereira_distill_2023-1, address = {Valencia, Spain}, series = {Proceedings of {Machine} {Learning} {Research}}, title = {Distill n' {Explain}: explaining graph neural networks using simple surrogates}, volume = {206}, shorttitle = {Distill n' {Explain}}, url = {https://proceedings.mlr.press/v206/pereira23a.html}, language = {en}, booktitle = {Proceedings of {The} 26th {International} {Conference} on {Artificial} {Intelligence} and {Statistics}}, publisher = {PMLR}, author = {Pereira, Tamara and Nascimento, Erik and Resck, Lucas E. and Mesquita, Diego and Souza, Amauri}, month = apr, year = {2023}, pages = {6199--6214}, bibtex_show = true, google_scholar_id = {UebtZRa9Y70C}, }
2021
- BSc thesis
Inferring and Explaining Potential Citations to Binding Precedents in Brazilian Supreme Court DecisionsLucas Emanuel Resck DominguesFundação Getulio Vargas , Dec 2021The Brazilian Supreme Court (STF) is the highest law court in Brazil and it is primarily responsible for guarding the Brazilian Constitution. To reduce judicial insecurity and the high Court’s workload, a Constitutional Amendment from 2004 allowed STF to create binding precedents (“Súmulas Vinculantes,” BPs). A BP is a statement that consolidates the understanding of STF about a legal matter and has mandatory application for lower branches of the Judiciary. Frequently, an STF Justice cites a BP in a decision, and it is trivial to search for these explicit citations using regular expressions. However, it is not trivial to assert whether a decision potentially cites the statement, in the sense of “it should have cited it, but it did not” or “it addresses a similar issue, so they are related.” This work explores machine learning and natural language processing (NLP) algorithms to infer and explain these potential citations. The inference is performed using models from classical machine learning theory and recent NLP research, and the explanation is achieved using a machine learning explainability technique. The models learn what characterizes a citation through training on documents with explicit citations, in which we demonstrate they achieve high performance. We present two case studies that demonstrate the usefulness of the trained models to search for potential citations when accompanied by the explainability technique to inform the most relevant parts of the document for the potential citation assignment.
@phdthesis{domingues_inferring_2021, address = {Rio de Janeiro, Brazil}, title = {Inferring and {Explaining} {Potential} {Citations} to {Binding} {Precedents} in {Brazilian} {Supreme} {Court} {Decisions}}, url = {https://hdl.handle.net/10438/31845}, language = {eng}, school = {Fundação Getulio Vargas}, author = {Domingues, Lucas Emanuel Resck}, month = dec, year = {2021}, bibtex_show = true, google_scholar_id = {_FxGoFyzp5QC}, }
2018
- ReportCircuits for Driving Low Power Direct Current MotorsLucas Emanuel Resck Domingues, and Júlia Gandini BlahunOriginal title in Portuguese: “Circuitos para Acionamento de Motores de Corrente Contínua de Baixa Potência” , Dec 2018
@techreport{domingues_circuits_2018, address = {Varginha, Brazil}, type = {Scientific initiation project report}, title = {Circuits for {Driving} {Low} {Power} {Direct} {Current} {Motors}}, language = {pt}, institution = {Federal Center for Technological Education of Minas Gerais}, author = {Domingues, Lucas Emanuel Resck and Blahun, Júlia Gandini}, year = {2018}, pages = {32}, bibtex_show = true, google_scholar_id = {roLk4NBRz8UC} }
2016
- ReportBrazilian Robotics Olympiad – OBR’2016, Level II Practical ModalityJúlia Gandini Blahun, Luiza de Souza Pinto Regina, and Lucas Emanuel Resck DominguesOriginal title in Portuguese: “Olimpíada Brasileira de Robótica – OBR’2016, Modalidade Prática de Nível II” , Dec 2016
@techreport{blahun_brazilian_2016, address = {Varginha, Brazil}, type = {Scientific initiation project report}, title = {Brazilian {Robotics} {Olympiad} – {OBR}’2016, {Level} {II} {Practical} {Modality}}, language = {pt}, institution = {Federal Center for Technological Education of Minas Gerais}, author = {Blahun, Júlia Gandini and Regina, Luiza de Souza Pinto and Domingues, Lucas Emanuel Resck}, year = {2016}, pages = {13}, bibtex_show = true, google_scholar_id = {LkGwnXOMwfcC} }