↓Skip to main content
  1. PaperReading/
  2. Nature/

KAD: Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images

Nature Communication

code

f3d2579e-bd75-415b-a5e0-8a10b01473cb

  1. Knowledge database
    1. knowledge graph: concept-relation-concept triplets
    2. concept info list: concept-definition pairs
  2. Pre-training
    1. Text contrastive learning between medical concepts or definitions
    2. image-text contrastive learning with paired chest X-rays and extracted entities(by tools like ChatGPT)
    3. optimize a Disease Query Network (DQN) for classification
  3. Zero-shot inference
    1. encode the disease name as a query input, and DQN will output the probability that the pathology is present in the input image