GNSS & Machine Learning Engineer

Tag: NLP (Page 1 of 2)

Flan-U-PaLM: Google presents better language models without massive compute

Google presents two complementary techniques to significantly improve language models without massive extra compute:

UL2R (UL2 Repair): additional stage of continued pre-training with the UL2 (Unified Language Learner) objective (paper) for training language models as denoising tasks, where the model has to recover missing sub-sequences of a given input. Applying it to PaLM results in new language model U-PaLM.

Flan (Fine-tuning language model): instruction fine-tuning on a collection of NLP datasets. Applying it to PaLM results in language model Flan-PaLM.

Combination of two approaches applied to PaLM results in Flan-U-PaLM.

OpenAI released new GPT-3 model text-davinci-003

OpenAI released its new GPT-3 model text-davinci-003. After first tests it seems to be shockingly good.

Here is an example:

Write a poem about working as a GNSS engineer.

A GNSS engineer who works with a passion
Focused on quality and precision
A job that demands great dedication

To help others with navigation

The engineer’s purpose is so great
Measuring and mapping in space
Collecting data with all his might
To make the world a better sight

Determining points with great accuracy
And measuring distances with agility
Anywhere in the world, the data is clear
The world travels of this engineer

From the satellites in the sky
A network of coordinates is what he relies
His work is vital, that’s no doubt
To scientists, architects, and surveyors throughout

Answering questions of direction and height
Mapping the world with great light
A profession that’s so much more
Than many people think before

The GNSS engineer works with precision
His work is a beautiful mission
To measure and map the world we see
His work is essential to the GNSS industry

Meta AI presents CICERO, the first AI to achieve human-level performance in strategy game Diplomacy

Meta AI presents CICERO, an AI agent that can negotiate and cooperate with people. It is the first AI system that achieves human-level performance in the popular strategy game Diplomacy. Cicero ranked in the top 10 of participants on

Yannic Kilcher gives a great discussion of the accompanying Science paper. A second paper is freely available on arXiv. The source code is accessible on GitHub.

Galactica: Paper Generator by Meta AI

Meta AI publishes with a large language model trained on scientific papers that allows to write a literature review, wiki article, or lecture note with references, formulas, etc. just by giving some text input about a topic. Even the paper about Galactica was written with the help of Galactica.

Just after a day, the webpage is now down. But the source code is available on GitHub. Yannic Kilcher made a nice paper review about Galactica where he also explains why the demo webpage has been taken down.

Self-ask improves over chain of thought prompting

The new self-ask prompting technique [Paper] lets the LLM (Large Language Model) formulate sub-questions to a given complex question, answers these simpler questions and finally combines the answers to the answer of the complex question. This technique already improves over chain of thought (CoT) prompting [1][2][3] (but no comparison to self-consistency) . However, with a few lines of source code [GitHub], it is also possible to let the simpler questions be answered by a Google search. In this way also questions relying on facts beyond the training corpus of the LLM can be answered!

Meanwhile people also started to combine LLMs not just with a Google search but also with a Python interpreter. Key tool in this context that allows to build up chains of actions is the Python package LangChain that also already integrates the self-ask technique. See also Dust for reproducing OpenAI’s WebGPT via advanced prompting and Everyprompt for parameterized text prompts within a playground application.

« Older posts

© 2022 Stephan Seeger

Theme by Anders NorenUp ↑