#vlms

Voltron: Legendary Mahou Shoujo (x)

Voltron: Legendary Mahou Shoujo (x)

Understanding sarcasm is a uniquely human challenge, requiring nuanced interpretation of both language and context—often visual cues. Recent advancements in vision-language models (VLMs) present an intriguing opportunity to automate this complex task. A new study explores how effectively these open-source AI models can detect multimodal sarcasm – that is, sarcasm conveyed through the interplay of…

The Rise of VLM-Powered Web Agents Vision-Language Models (VLMs) are rapidly transforming how we interact with the web, and this shift brings both exciting opportunities and new vulnerabilities. These models combine visual perception and natural language understanding, enabling powerful applications like content recommendation and product ranking. Consequently, recent research highlights the…

The field of artificial intelligence continues to evolve at a rapid pace, and recent advancements in vision-language models (VLMs) are reshaping how we approach complex tasks. These powerful AI systems are demonstrating remarkable capabilities across various domains. However, a concerning study published on arXiv highlights potential dangers when these innovative tools intersect with the legal…

The NanoVLM initiative develops VLMs for NVIDIA Jetson devices, specifically the Orin Nano. These models aim to improve interaction performance by increasing processing speed and decreasing memory usage. Documentation includes supported VLM families, benchmarks, and setup parameters such Jetson device and Jetpack compatibility. Video sequence processing, live streaming analysis, and multimodal chat via online user interfaces or command-line interfaces are also covered.

What's nanoVLM?

NanoVLM is the fastest and easiest repository for training and optimising micro VLMs.

Hugging Face streamlined this teaching method. We want to democratise vision-language model creation via a simple PyTorch framework. Inspired by Andrej Karratha's nanoGPT, NanoVLM prioritises readability, modularity, and transparency without compromising practicality. About 750 lines of code define and train nanoVLM, plus parameter loading and reporting boilerplate.

Architecture and Components

NanoVLM is a modular multimodal architecture with a modality projection mechanism, lightweight language decoder, and vision encoder. The vision encoder uses transformer-based SigLIP-B/16 for dependable photo feature extraction.

Visual backbone translates photos into language model-friendly embeddings.

Textual side uses SmolLM2, an efficient and clear causal decoder-style converter.

Vision-language fusion is controlled by a simple projection layer that aligns picture embeddings into the language model's input space.

Transparent, readable, and easy to change, the integration is suitable for rapid prototyping and instruction.

The effective code structure includes the VLM (~100 lines), Language Decoder (~250 lines), Modality Projection (~50 lines), Vision Backbone (~150 lines), and a basic training loop (~200 lines).

Sizing and Performance

HuggingFaceTB/SmolLM2-135M and SigLIP-B/16-224-85M backbones create 222M nanoVLMs. Version nanoVLM-222M is available.

NanoVLM is compact and easy to use but offers competitive results. The 222M model trained for 6 hours on a single H100 GPU with 1.7M samples from the_cauldron dataset had 35.3% accuracy on the MMStar benchmark. SmolVLM-256M-like performance was achieved with fewer parameters and computing.

NanoVLM is efficient enough for educational institutions or developers using a single workstation.

Key Features and Philosophy

NanoVLM is a simple yet effective VLM introduction.

It enables users test micro VLMs' capabilities by changing settings and parameters.

Transparency helps consumers understand logic and data flow with minimally abstracted and well-defined components. This is ideal for repeatability research and education.

Its modularity and forward compatibility allow users to replace visual encoders, decoders, and projection mechanisms. This provides a framework for multiple investigations.

Get Started and Use

Cloning the repository and establishing the environment lets users start. Despite pip, uv is recommended for package management. Dependencies include torch, numpy, torchvision, pillow, datasets, huggingface-hub, transformers, and wandb.

NanoVLM includes easy methods for loading and storing Hugging Face Hub models. VisionLanguageModel.from_pretrained() can load pretrained weights from Hub repositories like “lusxvr/nanoVLM-222M”.

Pushing trained models to the Hub creates a model card (README.md) and saves weights (model.safetensors) and configuration (config.json). Repositories can be private but are usually public.

Model can load and store models locally.VisionLanguageModel.from_pretrained() and save_pretrained() with local paths.

To test a trained model, generate.py is provided. An example shows how to use an image and “What is this?” to get cat descriptions.

In the Models section of the NVIDIA Jetson AI Lab, “NanoVLM” is included, however the content focusses on using NanoLLM to optimise VLMs like Llava, VILA, and Obsidian for Jetson devices. This means Jetson and other platforms can benefit from nanoVLM's small VLM optimisation techniques.

Training

Train nanoVLM with the train.py script, which uses models/config.py. Logging with WANDB is common in training.

VRAM specs

VRAM needs must be understood throughout training.

A single NVIDIA H100 GPU evaluating the default 222M model shows batch size increases peak VRAM use.

870.53 MB of VRAM is allocated after model loading.

Maximum VRAM used during training is 4.5 GB for batch size 1 and 65 GB for batch size 256.

Before OOM, 512-batch training peaked at 80 GB.

Results indicate that training with a batch size of up to 16 requires at least ~4.5 GB of VRAM, whereas training with a batch size of up to 16 requires roughly 8 GB.

Variations in sequence length or model architecture affect VRAM needs.

To test VRAM requirements on a system and setup, measure_vram.py is provided.

Contributions and Community

NanoVLM welcomes contributions.

Contributions with dependencies like transformers are encouraged, but pure PyTorch implementation is preferred. Deep speed, trainer, and accelerate won't work. Open an issue to discuss new feature ideas. Bug fixes can be submitted using pull requests.

Future research includes data packing, multi-GPU training, multi-image support, image-splitting, and VLMEvalKit integration. Integration into the Hugging Face ecosystem allows use with Transformers, Datasets, and Inference Endpoints.

In summary

The 3-year LLB program is a postgraduate legal education pathway designed for graduates who wish to enter the legal profession. Recognized by the Bar Council of India (BCI), it is a rigorous course that combines theoretical legal studies with practical training, equipping students with the skills necessary to excel in various legal careers.

Overview of the 3-Year LLB Program

The 3-year LLB is a concentrated program tailored for those who have already earned a bachelor's degree in any discipline. It provides an in-depth understanding of legal concepts, systems, and professional ethics, along with opportunities to gain hands-on experience through internships, moot courts, and research projects. The program typically spans six semesters over three years.

Eligibility Criteria

To pursue the 3-year LLB program, candidates must meet specific eligibility requirements:

Academic Qualification: A bachelor's degree in any field from a recognized university.

Minimum Marks: A minimum aggregate score of 45% to 50%, depending on the institution.

Entrance Exams: Some institutions require candidates to clear entrance tests such as CLAT PG, LSAT India, or university-specific exams.

Curriculum and Structure

The curriculum of the 3-year LLB program strikes a balance between foundational legal studies and specialization. It covers:

Core Subjects: These include Constitutional Law, Criminal Law, Contract Law, Family Law, and Torts.

Electives: Students may explore specialized fields such as Intellectual Property Law, Environmental Law, Cyber Law, and Corporate Law.

Practical Training: Moot court sessions, internships, and participation in legal aid clinics form a crucial part of the program, offering real-world exposure.

Research and Projects: Assignments and dissertations enable students to develop critical thinking and research skills.

Benefits of the 3-Year LLB Program

Accelerated Path to Legal Careers: Unlike the 5-year integrated law program, this course is ideal for graduates, offering a faster route to professional practice.

Comprehensive Skill Development: The program’s blend of theory and practical training ensures students are well-prepared for diverse legal roles.

Cost-Effective: It is generally more affordable compared to integrated law courses.

Diverse Career Opportunities: Graduates can pursue roles in litigation, corporate law, public policy, academia, or judiciary.

Career Opportunities

The 3-year LLB degree opens doors to a variety of career paths:

Litigation: Graduates can practice as advocates in courts after enrolling with the Bar Council of India.

Corporate Law: Many organizations hire legal advisors, compliance officers, or in-house counsel.

Judicial Services: Those interested in the judiciary can appear for judicial service exams to become judges.

Academics and Research: Advanced studies like LLM or Ph.D. can lead to careers in teaching and legal research.

Public Sector: Opportunities include roles in government legal departments and public sector undertakings.

Conclusion