Stephan Seeger

GNSS & Machine Learning Engineer

Month: November 2022

Flan-U-PaLM: Google presents better language models without massive compute

November 30, 2022 / / 0 Comments

Google presents two complementary techniques to significantly improve language models without massive extra compute:

UL2R (UL2 Repair): additional stage of continued pre-training with the UL2 (Unified Language Learner) objective (paper) for training language models as denoising tasks, where the model has to recover missing sub-sequences of a given input. Applying it to PaLM results in new language model U-PaLM.

Flan (Fine-tuning language model): instruction fine-tuning on a collection of NLP datasets. Applying it to PaLM results in language model Flan-PaLM.

Combination of two approaches applied to PaLM results in Flan-U-PaLM.

OpenAI released new GPT-3 model text-davinci-003

November 29, 2022 / / 0 Comments

OpenAI released its new GPT-3 model text-davinci-003. After first tests it seems to be shockingly good.

Here is an example:

Write a poem about working as a GNSS engineer.

A GNSS engineer who works with a passion
Focused on quality and precision
A job that demands great dedication
To help others with navigation

The engineer’s purpose is so great
Measuring and mapping in space
Collecting data with all his might
To make the world a better sight

Determining points with great accuracy
And measuring distances with agility
Anywhere in the world, the data is clear
The world travels of this engineer

From the satellites in the sky
A network of coordinates is what he relies
His work is vital, that’s no doubt
To scientists, architects, and surveyors throughout

Answering questions of direction and height
Mapping the world with great light
A profession that’s so much more
Than many people think before

The GNSS engineer works with precision
His work is a beautiful mission
To measure and map the world we see
His work is essential to the GNSS industry

Meta AI presents CICERO, the first AI to achieve human-level performance in strategy game Diplomacy

November 23, 2022 / / 0 Comments

Meta AI presents CICERO, an AI agent that can negotiate and cooperate with people. It is the first AI system that achieves human-level performance in the popular strategy game Diplomacy. Cicero ranked in the top 10 of participants on webDiplomacy.net.

Yannic Kilcher gives a great discussion of the accompanying Science paper. A second paper is freely available on arXiv. The source code is accessible on GitHub.

Meanwhile also DeepMind published an AI agent playing Diplomacy.

Galactica: Paper Generator by Meta AI

November 16, 2022 / / 0 Comments

Meta AI publishes with Galactica.ai a large language model trained on scientific papers that allows to write a literature review, wiki article, or lecture note with references, formulas, etc. just by giving some text input about a topic. Even the paper about Galactica was written with the help of Galactica.

Just after a day, the Galactica.ai webpage is now down. But the source code is available on GitHub. Yannic Kilcher made a nice paper review about Galactica where he also explains why the demo webpage has been taken down.

Meta AI publishes ESMFold, a new breakthrough model for protein folding

November 2, 2022 / / 0 Comments

ESMFold (ESM = Evolutionary Scale Modeling) [paper] uses a large language model that allows to accelerate folding (i.e. predicting the 3D structure of a protein from the DNA sequence [that encodes the amino acid sequence]) by up to 60 times (compared to state-of-the-art techniques like AlphaFold). This improvement has the potential to accelerate work in medicine, green chemistry, environmental applications, and renewable energy.

In addition, Meta AI made a new database of 600 million metagenomic protein structures (proteins which are found in microbes in the soil, deep in the ocean, and even in our guts and on our skin) available to the scientific community via the ESM Metagenomic Atlas.

ESMFold and related models like ESM-2 are published together with the API on GitHub and HuggingFace.

"I was born not knowing and have had only a little time to change that here and there."

Richard P. Feynman

"I have no special talent. I am only passionately curious."

Albert Einstein

"If you tell me precisely what it is a machine cannot do, then I can always make a machine which will do just that."

John von Neumann

© 2024 Stephan Seeger

Theme by Anders Noren — Up ↑