Question 1
An AI engineering team is building a next-generation machine-translation engine for enterprise and regulatory use, where accurately interpreting long, complex sentences is essential. The previous RNN-based model translated long sentences poorly because it handled words sequentially. To resolve these problems, the team decides that the new architecture should use self-attention mechanisms to process the full data sequence in parallel and effectively handle long-range dependencies.
Which deep learning architecture should the team choose to satisfy these requirements?
Legacy sequence-to-sequence models utilized Recurrent Neural Networks (RNNs) or Long Short-Term Memory (LSTM) networks, which process sequence data iteratively step-by-step. This sequential processing creates a severe computational bottleneck and suffers from vanishing or exploding gradients, which limits their capability to maintain semantic dependencies across long textual sequences. Transformer networks overcome these limitations by replacing recurrent recurrence entirely with multi-head self-attention mechanisms. Self-attention enables the model to weigh the contextual relationship of every token against every other token in the input sequence concurrently, regardless of their relative distance. This non-sequential architecture allows full-sequence parallel processing across distributed GPU/TPU clusters, drastically reducing training latency while preserving deep semantic context across complex enterprise and legal documents. In contrast, Convolutional Neural Networks (CNNs) excel in local spatial grid extraction (vision), Autoencoders specialize in unsupervised latent space compression and reconstruction, and Radial Basis Function (RBF) networks operate as localized function approximators. For state-of-the-art enterprise machine translation demanding long-range dependency resolution, Transformers serve as the definitive standard.
Reference Topics from Study Guide: Module 01: AI Foundations and Technology Ecosystem --- Deep Learning Architectures, Recurrent Mechanisms vs. Attention-Based Transformer Networks.