Show HN: KARMA – An evaluation framework for Medical AI systems

KARMA-OpenMedEvalKit is an expandable toolkit for assessing AI models in medical applications, featuring multiple healthcare-focused datasets with particular emphasis on the Indian healthcare environment. KARMA can evaluate text, image, and audio-based medical AI models using 21+ healthcare datasets We support popular models (Qwen, MedGemma, IndicConformer, OpenAI, Anthropic models - via AWS Bedrock, and practically any HuggingFace models) out-of-the-box KARMA also handles medical-specific evaluation needs like ASR models that need language-aware post-processing, or having LLM as a judge on rubric based evaluations. KARMA caches model outputs so you can iterate on metrics without re-running expensive inference. Medical AI evaluation is currently fragmented – researchers often build custom evaluation scripts for each project. KARMA provides standardized metrics and a registry system where you can easily plug in your own models and datasets. KARMA has extensible registry system with deco