Adapting Language Models to Specialized Texts

I was a developer of SciBERT, one of the first pretrained language models for scientific text. Our follow-on work on domain adaptation via continued pretraining won an honorable mention for best paper at ACL 2020 🏆.

I also develop methods for infusing language models with visual layout. I packaged these models into PaperMage, an open-source Python library that won best paper at ACL 2023 System Demos 🏆.

Since 2023, I’ve been working on out-of-domain generalization via parameter-efficient training and data augmentation.