NVIDIA Unveils TensorRT for RTX to Boost AI Application Performance

NVIDIA has announced the release of TensorRT for RTX, a new software development kit (SDK) designed to enhance the performance of AI applications on NVIDIA RTX GPUs. This SDK, which can be integrated into C++ and Python applications, is available for both Windows and Linux platforms. The announcement was made at the Microsoft Build event, highlighting the SDK’s potential to streamline high-performance AI inference across various workloads such as convolutional neural networks, speech models, and diffusion models, according to NVIDIA’s official blog.

Key Features and Benefits

TensorRT for RTX is positioned as a drop-in replacement for the existing NVIDIA TensorRT inference library, simplifying the deployment of AI models on NVIDIA RTX GPUs. It introduces a Just-In-Time (JIT) optimizer in its runtime, enhancing inference engines directly on the user’s RTX-accelerated PC. This innovation eliminates lengthy pre-compilation steps, improving application portability and runtime performance. The SDK supports lightweight application integration, making it suitable for memory-constrained environments with its compact size, under 200 MB.

The SDK package includes support for both Windows and Linux, C++ development header files, Python bindings for rapid prototyping, an optimizer and runtime library for deployment, a parser library for importing ONNX models, and various developer tools to simplify deployment and benchmarking.

Advanced Optimization Techniques

TensorRT for RTX applies optimizations in two phases: Ahead-Of-Time (AOT) optimization and runtime optimization. During AOT, the model graph is improved and converted to a deployable engine. At runtime, the JIT optimizer specializes the engine for execution on the installed RTX GPU, allowing for rapid engine generation and improved performance.

Notably, TensorRT for RTX introduces dynamic shapes, enabling developers to defer specifying tensor dimensions until runtime. This feature allows for flexibility in handling network inputs and outputs, optimizing engine performance based on specific use cases.

Enhanced Deployment Capabilities

The SDK also features a runtime cache for storing JIT-compiled kernels, which can be serialized for persistence across application invocations, reducing startup time. Additionally, TensorRT for RTX supports AOT-optimized engines that are runnable on NVIDIA Ampere, Ada, and Blackwell generation RTX GPUs, without requiring a GPU for building.

Moreover, the SDK allows for the creation of weightless engines, minimizing application package size when weights are shipped alongside the engine. This feature, along with the ability to refit weights during inference, provides developers greater flexibility in deploying AI models efficiently.

With these advancements, NVIDIA aims to empower developers to create real-time, responsive AI applications for various consumer-grade devices, enhancing productivity in creative and gaming applications.

Image source: Shutterstock