#tensorrt

Rekor Uses NVIDIA Technology for Traffic Relief and Roadway Safety as Texas Takes in More Residents.

For Texas and Philadelphia highways, the company is using AI-driven analytics utilizing NVIDIA AI, Metropolis, and Jetson, which might lower fatalities and enhance quality of life.

Jobs, comedy clubs, music venues, barbecues, and more are all attracting visitors to Austin. Traffic congestion, however, are a major city blues that have come with this growth.

Due to the surge of new inhabitants moving to Austin, Rekor, which provides traffic management and public safety analytics, gets a direct view of the growing traffic. To assist alleviate the highway issues, Rekor collaborates with the Texas Department of Transportation, which is working on a $7 billion initiative to remedy this.

Based in Columbia, Maryland, Rekor has been using NVIDIA Jetson Xavier NX modules for edge AI and NVIDIA Metropolis for real-time video understanding in Texas, Florida, Philadelphia, Georgia, Nevada, Oklahoma, and many other U.S. locations, as well as Israel and other countries.

Metropolis is a vision AI application framework for creating smart infrastructure. Its development tools include the NVIDIA DeepStream SDK, TAO Toolkit, TensorRT, and NGC catalog pretrained models. The tiny, powerful, and energy-efficient NVIDIA Jetson accelerated computing platform is ideal for embedded and robotics applications.

Rekor’s initiatives in Texas and Philadelphia to use AI to improve road management are the most recent chapter in a long saga of traffic management and safety.

Reducing Rubbernecking, Pileups, Fatalities and Jams

Rekor Command and Rekor Discover are the two primary products that Rekor sells. Traffic control centers can quickly identify traffic incidents and areas of concern using Command, an AI-driven software. It provides real-time situational awareness and notifications to transportation authorities, enabling them to maintain safer and less congested municipal roads.

Utilizing Rekor’s edge technology, discover completely automates the collection of thorough vehicle and traffic data and offers strong traffic analytics that transform road data into quantifiable, trustworthy traffic information. Departments of transportation may better plan and carry out their next city-building projects by using Rekor Discover, which gives them a comprehensive picture of how cars travel on roads and the effect they have.

Command has been spread around Austin by the corporation to assist in problem detection, incident analysis, and real-time response to traffic activities.

Rekor Command receives a variety of data sources, including weather, linked vehicle information, traffic camera video, construction updates, and data from third parties. After that, it makes links and reveals abnormalities, such as a roadside incident, using AI. Traffic management centers receive the data in processes for evaluation, verification, and reaction.

As part of the NVIDIA AI Enterprise software platform, Rekor is embracing NVIDIA’s full-stack accelerated computing for roadway intelligence and investing heavily in NVIDIA AI and NVIDIA AI Blueprints, reference workflows for generative AI use cases constructed with NVIDIA NIM microservices. NVIDIA NIM is a collection of user-friendly inference microservices designed to speed up foundation model installations on any cloud or data center while maintaining data security.

Rekor is developing AI agents for municipal services, namely in areas like traffic control, public safety, and infrastructure optimization, leveraging the NVIDIA AI Blueprint for video search and summarization. In order to enable a variety of interactive visual AI agents that can extract complicated behaviors from vast amounts of live or recorded video, NVIDIA has revealed a new AI blueprint for video search and summarization.

Philadelphia Monitors Roads, EV Charger Needs, Pollution

The Philadelphia Industrial Development Corporation (PIDC), which oversees the Philadelphia Navy Yard, a famous tourist destination, has difficulties managing the roads and compiling information on new constructions. According to a $6 billion rehabilitation proposal, the Navy Yard property will bring thousands of inhabitants and 12,000 jobs with over 150 firms and 15,000 workers on 1,200 acres.

PIDC sought to raise awareness of how road closures and construction projects influence mobility and how to improve mobility during major events and projects. PIDC also sought to improve the Navy Yard’s capacity to measure the effects of speed-mitigating devices placed across dangerous sections of road and comprehend the number and flow of car carriers or other heavy vehicles.

In order to handle any fluctuations in traffic, Discover offered PIDC information about further infrastructure initiatives that must be implemented.

By knowing how many electric cars are coming into and going out of the Navy Yard, PIDC can make informed decisions about future locations for the installation of EV charging stations. Navy Yard can better plan possible locations for EV charge station deployment in the future by using Rekor Discover, which gathers data from Rekor’s edge systems which are constructed with NVIDIA Jetson Xavier NX modules for powerful edge processing and AI to understand the number of EVs and where they’re entering and departing.

By examining data supplied by the AI platform, Rekor Discover allowed PIDC planners to produce a hotspot map of EV traffic. The solution uses Jetson and NVIDIA’s DeepStream data pipeline for real-time traffic analysis. To further improve LLM capabilities, it makes advantage of NVIDIA Triton Inference Server.

The PIDC sought to reduce property damage and address public safety concerns about crashes and speeding. When average speeds are higher than what is recommended on certain road segments, traffic calming measures are being implemented using speed insights.

NVIDIA Jetson Xavier NX to Monitor Pollution in Real Time

Rekor’s vehicle identification models, which were powered by NVIDIA Jetson Xavier NX modules, were able to follow pollution to its origins, moving it one step closer to mitigation than the conventional method of using satellite data to attempt to comprehend its placements.

In the future, Rekor is investigating the potential applications of NVIDIA Omniverse for the creation of digital twins to model traffic reduction using various techniques. Omniverse is a platform for creating OpenUSD applications for generative physical AI and industrial digitization.

Creating digital twins for towns using Omniverse has significant ramifications for lowering traffic, pollution, and traffic fatalities all of which Rekor views as being very advantageous for its clients.

NVIDIA TensorRT

Thanks to NVIDIA RTX and GeForce RTX technology, the AI PC age is arrived. Along with it comes a new language that can be difficult to understand when deciding between the many desktop and laptop options, as well as a new method of assessing performance for AI-accelerated tasks. This article is a part of the AI Decoded series, which shows off new RTX PC hardware, software, tools, and accelerations while demystifying AI by making the technology more approachable.

While frames per second (FPS) and related statistics are easily understood by PC gamers, new measures are needed to measure AI performance.

Emerging as the Best

Trillions of operations per second, or TOPS, is the initial baseline. The key term here is trillions; the processing power required for generative AI jobs is truly enormous. Consider TOPS to be a raw performance indicator, like to the horsepower rating of an engine.

Take Microsoft’s recently unveiled Copilot+ PC series, for instance, which has neural processing units (NPUs) capable of up to 40 TOPS. For many simple AI-assisted tasks, such as asking a nearby chatbot where yesterday’s notes are, 40 TOPS is sufficient.

However, a lot of generative AI tasks are more difficult. For all generative tasks, the NVIDIA RTX and GeForce RTX GPUs offer performance never seen before; the GeForce RTX 4090 GPU offers more than 1,300 TOPS. AI-assisted digital content production, AI super resolution in PC gaming, image generation from text or video, local large language model (LLM) querying, and other tasks require processing power comparable to this.

Put in Tokens to Start Playing

TOPS is just the start of the tale. The quantity of tokens produced by the model serves as a gauge for LLM performance.

The LLM’s output is tokens. A word in a sentence or even a smaller piece like whitespace or punctuation might serve as a token. The unit of measurement for AI-accelerated task performance is “tokens per second.”

Batch size, or the quantity of inputs processed concurrently in a single inference pass, is another crucial consideration. The ability to manage many inputs (e.g., from a single application or across multiple apps) will be a critical distinction, as an LLM will be at the basis of many modern AI systems. Greater batch sizes demand more memory even though they perform better for concurrent inputs, particularly when paired with larger models.

NVIDIA TensorRT-LLM

Because of their massive amounts of dedicated video random access memory (VRAM), Tensor Cores, and TensorRT-LLM software, RTX GPUs are incredibly well-suited for LLMs.

High-speed VRAM is available on GeForce RTX GPUs up to 24GB and on NVIDIA RTX GPUs up to 48GB, allowing for larger models and greater batch sizes. Additionally, RTX GPUs benefit from Tensor Cores, which are specialised AI accelerators that significantly accelerate the computationally demanding tasks necessary for generative AI and deep learning models. Using the NVIDIA TensorRT software development kit (SDK), which enables the highest-performance generative AI on the more than 100 million Windows PCs and workstations powered by RTX GPUs, an application can quickly reach that maximum performance.

RTX GPUs achieve enormous throughput benefits, particularly as batch sizes increase, because to the combination of memory, specialised AI accelerators, and optimised software.

Text to Image More Quickly Than Before

Performance can also be assessed by measuring the speed at which images are generated. Stable Diffusion, a well-liked image-based AI model that enables users to quickly translate text descriptions into intricate visual representations, is one of the simplest methods.

Users may easily build and refine images from text prompts to get the desired result with Stable Diffusion. These outcomes can be produced more quickly when an RTX GPU is used instead of a CPU or NPU to process the AI model.

When utilising the TensorRT extension for the well-liked Automatic1111 interface, that performance increases even further. With the SDXL Base checkpoint, RTX users can create images from prompts up to two times faster, greatly simplifying Stable Diffusion operations.

TensorRT Acceleration

TensorRT acceleration was integrated to ComfyUI, a well-liked Stable Diffusion user interface, last week. Users of RTX devices may now create images from prompts 60% quicker, and they can even utilise TensorRT to transform these images to videos 70% faster utilising Stable Video Diffuson.

The new UL Procyon AI Image Generation benchmark tests TensorRT acceleration and offers 50% faster speeds on a GeForce RTX 4080 SUPER GPU than the quickest non-TensorRT implementation.

The much awaited text-to-image model from Stable Diffusion 3 Stability AI will soon receive TensorRT acceleration, which will increase performance by 50%. Furthermore, even more performance acceleration is possible because to the new TensorRT-Model Optimizer. This leads to a 50% decrease in memory use and a 70% speedup over the non-TensorRT approach.

Naturally, the actual test is found in the practical application of refining an initial prompt. By fine-tuning prompts on RTX GPUs, users may improve image production much more quickly it takes seconds instead of minutes when using a Macbook Pro M3 Max. When running locally on an RTX-powered PC or workstation, users also benefit from speed and security with everything remaining private.

The Results Are Available and Can Be Shared

Recently, the open-source Jan.ai team of engineers and AI researchers integrated TensorRT-LLM into their local chatbot app, then put these optimisations to the test on their own system.Image Credit to NVIDIA

TensorRT-LLM

The open-source llama.cpp inference engine was utilised by the researchers to test TensorRT-LLM’s implementation on a range of GPUs and CPUs that the community uses. They discovered that TensorRT is more effective on consecutive processing runs and “30-70% faster than llama.cpp on the same hardware.” The group invited others to assess the performance of generative AI independently by sharing its methodology.

Discover RTX’s Chat’s Speed

A free, user-friendly chatbot demo that is tailored to your area is called Chat with RTX. It is constructed using RTX acceleration, TensorRT-LLM, and RAG functionality. It is compatible with many open-source LLMs, including as Mistral’s Mistral and Meta’s Llama 2. In a later version, support for Google’s Gemma will arrive.

This article is a part of the NVIDIA AI Decoded series, which showcases new RTX PC and workstation accelerations and hardware while attempting to demystify AI by making the concept more approachable. Chatbots are among the first widely used applications of AI, whether it is experiencing an iPhone moment.

Large language models and deep learning algorithms that have been retrained on enormous datasets the size of the internet itself which are capable of text and other content generation, translation, summation, prediction, and recognition—are what enable them. On PCs and workstations with NVIDIA GeForce and RTX GPUs, they may operate locally.

Large text summaries, data classification and mining for insights, and producing new text in a user-specified style, tone, or format are among the many skills that LLMs excel in. They can help with communication in any language, even ones that are not spoken by people, such genetic, computer, or protein sequences.

Later generations of LLMs were trained on several kinds of data, although the first models only dealt with text. Images, music, videos, and other material types may be recognised and generated by these TensorRT-LLM.

Among the first to introduce LLMs to a consumer audience were chatbots such as ChatGPT, which had a recognisable UI designed to interact with and react to natural language inputs. Since then, LLMs have been utilised to support scientists working on drug and vaccine research as well as developers writing code.

AI Models With TensorRT-LLM

However, such functionalities rely heavily on computationally complex AI models. With RTX GPUs—which are designed specifically for AI—combined with sophisticated optimisation methods and algorithms like quantisation, it is possible to build LLMs small enough and PCs strong enough to operate locally without an internet connection. Furthermore, a new generation of thin LLMs, such as Mistral, one of the LLMs enabling Chat with RTX, paves the way for cutting-edge performance with less power and storage requirements.

Why Are LLMs Important?

Numerous sectors, use cases, and operations may be tailored for LLMs. Their high-speed performance and adaptability allow them to improve performance and efficiency on almost all language-based jobs.

LLMs are often used in language translation applications like DeepL, which provide accurate results using AI and machine learning.

To improve patient care, medical experts are using textbooks and other medical data to educate LLMs. Retailers are using chatbots with LLM capabilities to provide excellent customer service. LLMs are being used by financial analysts to summarise and transcribe earnings calls and other significant meetings. And that’s just the very beginning.

Writing assistants based on LLMs and chatbots like Chat with RTX are revolutionising every aspect of knowledge labor, from legal operations to content marketing and copy writing. One of the first LLM-powered products that hint at the AI-assisted software development future was coding helpers. These days, initiatives like ChatDev combine LLMs with AI agents—smart bots—that function independently to assist with question answering or carry out online chores in order to create a virtual software business that operates on demand. Simply inform the system of the kind of app required, and watch it go to work.

Simple as Starting a Discussion

A chatbot like ChatGPT, which streamlines the use of LLMs using plain language and reduces human involvement to as easy as instructing the model what to do, is often how many people first encountered generative AI.

LLM-powered chatbots may create creative poems, aid with vacation ideas, prepare emails to customer care, and even help with marketing material.

Technological developments in picture production and TensorRT-LLM have expanded the capabilities of chatbots to include image analysis and generation, all while preserving the delightfully simple user interface. Simply submit a picture and request that the system examine it, or provide the bot a description of the image. It’s still conversation, but with images now.

Future developments will enable LLMs to do more arithmetic, reasoning, logic, and other activities, enabling them to decompose complicated requests into smaller, more manageable jobs.

Additionally, work is being done on AI agents, which are programs that can take a complicated request, divide it up into smaller ones, then interact with LLMs and other AI systems on their own to finish tasks. An example of an AI agent framework is ChatDev, however agents aren’t only for technical jobs.

Use RAG to Unlock Data

Considerably though chatbots and TensorRT-LLM are quite effective in general, they may be considerably more beneficial when paired with user data specific to each person. By doing this, they may assist in summarising years’ worth of bank and credit card bills, searching through thick user manuals to discover the solution to a technical query concerning a piece of hardware, or analyzing email inboxes to identify patterns.

Optimizing LLMs for a certain dataset may be done with ease and effectiveness via retrieval-augmented generation, or RAG.

RAG incorporates data retrieved from outside sources to improve the precision and dependability of generative AI models. RAG allows users to communicate with data repositories and allows an LLM to mention its sources by linking it to almost any external site. All the user has to do to engage with the chatbot is point it in the direction of a file or directory.

For instance, a typical TensorRT-LLM will be broadly knowledgeable about marketing strategies, best practices for content development, and fundamental understanding of a certain sector or clientele. However, it might assess the content and assist in creating a customized approach if it were connected via RAG to marketing materials assisting in a product launch.

Any LLM may be used with RAG as long as the application supports it. The Chat with RTX tech demo from NVIDIA is an illustration of how RAG may link an LLM to a private dataset. On computers equipped with a GeForce RTX or NVIDIA RTX professional GPU, it operates locally.

Local files on a PC may be quickly and easily connected to a compatible TensorRT-LLM by dumping files into a folder and directing the demo to that folder. By doing this, it may provide prompt, contextually appropriate replies to questions.

Results are quick and user data remains on the device because Chat with RTX runs locally on Windows with GeForce RTX PCs and NVIDIA RTX workstations. Chat with RTX allows users to handle sensitive data locally on a PC without requiring an internet connection or sharing it with a third party, as opposed to depending on cloud-based services.