Introduction: Our teams at Tower Research Capital recently attended the 41st International Conference on Machine Learning, ICML 2024, in Vienna, Austria. We’re thrilled to have participated and been able to engage with the dynamic and ever-evolving ML community of talented individuals. From code automation to large language models (LLMs), time series analysis, and secure machine learning, the conference offered deep insights into the future of AI and ML.

Ramit, Tower’s Global Head of Core AI & ML, shares his thoughts from ICML 2024 below.

Code Agents & Automation

Advancements in code agents and automation were a major focus at ICML 2024. Of these, one notable contribution was NExT: Teaching Large Language Models to Reason About Code Execution. This approach pushes the boundaries of how LLMs interact with code, using chain-of-thought prompts to guide the model through reasoning steps. The result is a more sophisticated understanding of code execution, enabling these models to perform tasks like code repair with greater accuracy and autonomy.

Another intriguing development was “R2e: Turning Any GitHub Repository into a Programming Agent Environment”. This work focuses on creating interactive programming environments where LLMs can engage with existing codebases, enabling tasks like automated testing, code refactoring, and even feature development. By transforming repositories into dynamic environments, R2e opens up new possibilities for integrating AI directly into the software development lifecycle, enhancing productivity and reducing human error.

These advancements represent a significant leap forward in the capabilities of code agents, making them more powerful tools for automating and streamlining programming tasks. As these technologies continue to evolve, they promise to transform the way software is developed, tested, and maintained, pushing the boundaries of what can be achieved with AI-driven automation.

Large Language Models

LLMs continue to evolve, with a focus on making them more adaptable and efficient for a variety of tasks. Of these, one area of note was the Adaptive Feature Transfer (AFT) method, which offers a novel approach to knowledge transfer between models. Unlike traditional methods that rely on weight transfer, AFT operates by selectively transferring features that are most relevant to specific tasks. This approach not only improves the efficiency of smaller, fine-tuned models, but also enhances their performance on downstream tasks like text classification and sentiment analysis, where precision and adaptability are crucial. Additionally, FrameQuant introduces a flexible, low-bit quantization technique that enables large models, such as CodeLlama-70b, to be efficiently deployed on existing hardware with minimal loss in performance.

Time Series

Time series was another major focus at ICML 2024, with new methodologies pushing the boundaries of what’s possible in forecasting and representation learning. One of the standout models was TimeSFM, a decoder-only foundation model for time series forecasting inspired by the training regimes used in LLMs. TimeSFM demonstrated impressive zero-shot performance across various prediction horizons, making it particularly useful for applications like financial forecasting, where models must quickly adapt to new patterns without retraining. Another significant contribution was **TSLANet**, which rethinks how transformers are applied to time series data. TSLANet provides a more lightweight and efficient alternative to LSTM models, showing great promise in tasks that require fast, parallel processing, such as high-frequency trading and real-time anomaly detection. These advancements suggest that time series models are becoming more versatile and capable, enabling timelier and more accurate decision-making across a variety of domains.

Differential Privacy & Secure Models

Security and privacy in ML were also key themes at ICML 2024, with a particular focus on differential privacy (DP) and the challenges of training models on private, distributed data. One standout contribution was the introduction of Private Evolution-Text (PrE-Text), a method for generating differentially private synthetic textual data. This approach addresses the limitations of on-device training — such as limited computational resources and communication overhead — by enabling more efficient and secure training off-device. PrE-Text has been shown to outperform traditional on-device training across multiple datasets while using significantly fewer resources.

We participated in numerous other fascinating presentations at ICML 2024, far more than we could possibly cover in one summary. Advancements like these inspire us to stay focused on the future of machine learning, and we’re excited about the potential to contribute to these developments.

If you’re passionate about AI and ML and interested in exploring opportunities, we’re always looking for talented individuals to join our team. Reach out to our team to learn more about open roles at Tower!