GNSS & Machine Learning Engineer

Tag: OpenAI

OpenAI releases ChatGPT plugins

OpenAI announced on Mar 23, 2023, the availability of plugins within ChatGPT. Access is currently limited to ChatGPT Plus subscribers that joined a waitlist and have been selected by OpenAI.

Plugins can be automatically called by ChatGPT’s underlying LLM (Large Language Model, currently GPT-3.5 or GPT-4) in order to answer the questions of the user.

In order to make this work, plugins have to be registered in the ChatGPT user interface with a manifest file
(ai-plugin.json) that is hosted in the developer’s domain at
yourdomain.com/.well-known/ai-plugin.json. The file contains in a prescribed format
– metadata about the plugin (name, logo)
– details about the authentication mechanism
– an OpenAI specification for the endpoints of the API
– and a general description for the LLM of what the plugin can do.
The web API needs to define an endpoint “/.well-known/ai-plugin.json” to access the content of this file.

In addition to the manifest file, an openapi.yaml file, that defines the OpenAI specification, has to be generated that is referenced in the “api” section of the manifest file via the “url” field. This file contains a detailed description of the API endpoints. The web API needs to define an endpoint “/openapi.yaml” to access the content of this file.

When the user has activated a registered plugin and starts a conversation, the plugin’s description is injected into the message to ChatGPT, but invisible to the user. In this way, the LLM may choose an API call from the plugin if this seems relevant to the user’s question. The LLM will then incorporate the API result into the response to the user. More details can be found in OpenAI’s documentation.

Among the already available plugins, a few stand out. With the Wolfram plugin, all kinds of computational problems can be solved. And with the Zapier plugin, more than 5000 apps can be accessed. OpenAI itself introduced a web browser (that uses the Bing search API) and a code interpreter plugin (that runs in a sandbox without an internet connection). In addition, they open-sourced the code for a knowledge base retrieval plugin, that has to be self-hosted by a developer.

Interestingly enough, OpenAI notices that plugins will likely have wide-ranging societal implications and that language models with access to tools will likely have much greater economic impacts than those without. They expect the current wave of AI technologies to have a big effect on the pace of job transformation, displacement, and creation. OpenAI discusses the impact potential of large language models at the labor market in a recent publication.

Just a day after the OpenAI announcement of ChatGPT plugins, the open-source community already integrated these plugins also into LangChain. This is done just by referring to the plugin manifest file ai-plugin.json (see Twitter), e.g.:

tool = AIPluginTool.from_plugin_url( "https://www.clarna.com/.wellknown/ai-plugin.json")

All the other exciting news of the week is well summarized by Matt Wolfe (Google Bard, NVIDIA GTC, Adobe Firefly, Image Generation in Bing via DALL-E2, Microsoft Loop, AI in Canva, GitHub Copilot X, AI in Ubisoft, Metahuman by Unreal Engine).

OpenAI releases GPT-4

OpenAI released GPT-4 within ChatGPT on March 14, 2023, described in detail in a 98-pages paper (summarized on youtube).

  • Available to ChatGPT-Plus subscribers (currently with a cap that is changing over time, e.g. 100 messages every 4 hours, or 25 messages every 3 hours).
  • Still based on training data that cuts off Sept 2021.
  • It still does not learn from its experience.
  • Still no internet access.
  • The training was already finalized in Aug 2022.
  • Fine-tuned via RLHF (Reinforcement Learning with Human Feedback).
  • API waitlist is open (so no API access yet for everyone)
  • API prices (for comparison: GPT-3.5-turbo $0.002 per 1k tokens):
    • gpt-4: 8K context window (about 13 pages of text) will cost $0.03 per 1K prompt tokens and $0.06 per 1K completion tokens.
    • gpt-4-32k: 32K context window (about 52 pages of text) will cost $0.06 per 1K prompt tokens and $0.12 per 1K completion tokens.
  • The number of parameters and size of the training data set have both not been published. So competitors are not encouraged to replicate these performance ingredients but are referred to a freely available benchmark (OpenAI Evals) that measures the real performance.
  • GPT-4 ranks in the 10% best of the bar exam and 0.5% best of biology olympiad.
  • GPT-4 can handle contexts of over 25,000 words.
  • GPT-4 can access images as inputs and can generate captions, classifications, and analyses. However, this image-to-text functionality is not yet publicly available.
  • Microsoft Bing was already using an early version of GPT-4 in the last few weeks.

An excellent overview by Greg Brockman, President and co-founder of OpenAI, can be found on youtube.

Microsoft released Visual ChatGPT on March 08, 2023, in a paper and with source code on GitHub and Hugging Face. Although this does not seem to be GPT-4-based, it demonstrates similar image capabilities via a combination of pre-existing technologies (generate/modify [text-to-image], and describe [image-to-text]).

Two days after the GPT-4 release, Microsoft announced on March 16, 2023, the integration of GPT-4 into their Office products as a feature they called CoPilot. Copilot is not yet available for general use, but Microsoft plans to roll it out gradually to selected customers in the coming months.

OpenAI releases ChatGPT and Whisper APIs

On March 01, 2023, OpenAI announced the releases of APIs for ChatGPT (published on Nov 30, 2022) and the automatic speech recognition (ASR) engine Whisper for speech-to-text (STT) transcription (and translation) that was open-sourced in Sept 2022.

The ChatGPT model family is called gpt-3.5-turbo and costs just $0.002 per 1k tokens, which is 10 times cheaper than the existing GPT-3.5 models. Instead of consuming unstructured text as traditionally done by GPT, the ChatGPT models consume a sequence of messages with metadata following a new format called Chat Markup Language (ChatML). The number of tokens (tokens in prompt + tokens in response as available via response[‘usage’][‘total_tokens’]) is restricted to 4096. Notice that there is no possibility to fine-tune gpt-3.5-turbo models.

For Whisper the large-v2 model is now available through an API for a price of $0.006 per minute. The API contains endpoints for transcriptions (transcribes in source language) and translations (transcribes into English).

In addition, the possibility of dedicated instances for professional users was announced that can make economical sense beyond ~450M tokens per day.

A significant change that was made in the Terms of Service and Usage Polices is that data submitted to the API is no longer used for service improvements (e.g. model training) unless an organization opts in. Before it was necessary to opt-out.

OpenAI announced ChatGPT Plus

OpenAI announced on Feb 01, 2023, a new subscription plan, ChatGPT Plus, that will be available for $20/month. Benefits are

  • General access to ChatGPT, even during peak times
  • Faster response times
  • Priority access to new features and improvements

ChatGPT Plus will be available first just to customers in the United States.

Since Feb 10, 2023, it is also available in Germany.

On Feb 06, 2023, Google’s CEO Sundar Pichai announced with Bard a competitor to ChatGPT in a message to Google employees. Bard is based on LaMDA (Language Model for Dialog Applications) whose first version was unveiled on May 18, 2021.

Only 2 days later, on Feb 08, 2023, Microsoft’s CEO Satya Nadella demonstrated in a presentation the integration of ChatGPT into Microsoft Bing (codename Sydney).

Virtual Machine inside ChatGPT

People start to realize what is possible with ChatGPT and there are already some great summaries available [1][2] . However, the currently most exciting prompt I have seen is to let ChatGPT answer like being a Linux terminal. This effectively opens up a complete virtual machine inside ChatGPT by predicting the proper text answer on user input. Doing this in a recursive way feels mind-blowing …

Lots of people are now experimenting with this idea.

More info about ChatGPT can be found in this video by Yannic Kilcher.

OpenAI released new GPT-3 model text-davinci-003

OpenAI released its new GPT-3 model text-davinci-003. After first tests it seems to be shockingly good.

Here is an example:

Write a poem about working as a GNSS engineer.

A GNSS engineer who works with a passion
Focused on quality and precision
A job that demands great dedication

To help others with navigation

The engineer’s purpose is so great
Measuring and mapping in space
Collecting data with all his might
To make the world a better sight

Determining points with great accuracy
And measuring distances with agility
Anywhere in the world, the data is clear
The world travels of this engineer

From the satellites in the sky
A network of coordinates is what he relies
His work is vital, that’s no doubt
To scientists, architects, and surveyors throughout

Answering questions of direction and height
Mapping the world with great light
A profession that’s so much more
Than many people think before

The GNSS engineer works with precision
His work is a beautiful mission
To measure and map the world we see
His work is essential to the GNSS industry

© 2023 Stephan Seeger

Theme by Anders NorenUp ↑